Oct 30 00:31:40.477197 ip-10-0-130-245 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 00:31:40.477207 ip-10-0-130-245 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 00:31:40.477214 ip-10-0-130-245 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 00:31:40.477443 ip-10-0-130-245 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 00:31:50.505289 ip-10-0-130-245 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 00:31:50.505307 ip-10-0-130-245 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e1d505695e03478a80bce3f387eb528e --
Oct 30 00:33:55.007987 ip-10-0-130-245 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 00:33:55.386357 ip-10-0-130-245 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:33:55.386357 ip-10-0-130-245 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 00:33:55.386357 ip-10-0-130-245 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:33:55.386357 ip-10-0-130-245 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:33:55.391358 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391280    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:33:55.391358 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391358    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391362    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391366    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391369    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391372    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391375    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391378    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391381    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391383    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391386    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391389    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391392    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391395    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391397    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391400    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391403    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391411    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391414    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391417    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:33:55.391427 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391419    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391422    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391424    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391428    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391430    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391433    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391436    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391439    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391442    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391444    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391447    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391449    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391452    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391455    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391457    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391460    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391463    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391465    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391468    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:33:55.391868 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391470    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391472    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391476    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391480    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391483    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391486    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391489    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391492    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391494    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391497    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391500    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391502    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391505    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391507    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391510    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391514    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391517    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391520    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391522    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391525    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:33:55.392315 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391528    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391530    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391533    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391535    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391538    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391540    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391543    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391546    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391549    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391551    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391554    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391556    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391559    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391561    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391564    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391567    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391569    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391572    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391575    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391578    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:33:55.392794 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391581    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391583    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391586    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391589    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391591    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391594    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391596    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391599    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391602    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391605    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391607    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391610    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391613    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391616    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391618    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391621    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391625    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391629    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391631    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:33:55.393259 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391634    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391637    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391640    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391642    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391646    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391648    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391650    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.391653    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392401    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392408    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392412    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392414    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392417    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392420    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392423    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392425    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392428    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392430    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392433    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:33:55.393698 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392436    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392440    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392443    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392446    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392448    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392453    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392457    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392460    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392463    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392465    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392468    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392471    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392473    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392476    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392479    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392481    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392484    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392486    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392489    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:33:55.394161 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392492    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392494    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392497    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392499    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392502    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392504    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392507    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392510    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392512    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392515    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392517    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392520    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392522    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392525    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392529    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392532    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392534    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392537    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392539    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392542    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:33:55.394607 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392544    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392547    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392550    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392552    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392555    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392557    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392560    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392562    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392565    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392567    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392570    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392572    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392575    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392577    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392579    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392582    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392584    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392589    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392592    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:33:55.395092 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392596    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392599    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392601    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392604    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392606    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392609    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392611    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392615    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392618    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392621    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392624    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392626    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392629    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392632    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392634    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392637    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392640    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392642    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392645    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392648    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:33:55.395539 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392650    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392653    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392655    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392658    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392660    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392663    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392665    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392668    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392670    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392673    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392677    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392680    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392683    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392685    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392688    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392691    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.392693    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393222    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393229    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393232    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:33:55.396021 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393235    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393238    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393241    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393243    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393246    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393248    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393251    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393254    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393257    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393260    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393263    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393265    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393269    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393271    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393274    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393277    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393279    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393282    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393284    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:33:55.396513 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393287    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393289    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393292    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393294    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393297    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393299    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393302    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393305    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393308    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393311    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393313    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393316    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393319    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393321    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393324    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393327    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393329    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393331    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393334    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393337    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:33:55.396972 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393340    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393342    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393345    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393347    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393350    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393352    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393356    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393360    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393363    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393366    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393370    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393372    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393375    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393377    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393380    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393383    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393385    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393388    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393390    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393393    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:33:55.397440 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393396    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393399    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393401    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393404    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393407    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393410    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393412    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393415    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393419    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393421    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393435    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393439    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393442    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393445    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393447    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393450    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393454    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393457    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393461    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:33:55.397952 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393463    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393466    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393468    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393471    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393473    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393476    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393478    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393483    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393487    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393490    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393493    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393495    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393498    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393500    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393504    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393506    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393509    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393512    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393514    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393517    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:33:55.398418 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393519    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393522    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393524    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393527    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.393529    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393620    2581 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393627    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393633    2581 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393638    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393642    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393646    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393650    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393654    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393658    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393661    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393665    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393668    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393671    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393675    2581 flags.go:64] FLAG: --cgroup-root=""
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393679    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393682    2581 flags.go:64] FLAG: --client-ca-file=""
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393685    2581 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393688    2581 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 00:33:55.398886 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393692    2581 flags.go:64] FLAG: --cluster-domain=""
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393695    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393698    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393701    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393719    2581 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393724    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393727    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393730    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393734    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393737    2581 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393740    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393743    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393746    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393749    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393754    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393757    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393760    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393763    2581 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393766    2581 flags.go:64] FLAG: --enable-server="true"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393769    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393774    2581 flags.go:64] FLAG: --event-burst="100"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393777    2581 flags.go:64] FLAG: --event-qps="50"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393780    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393784    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 00:33:55.399440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393787    2581 flags.go:64] FLAG: --eviction-hard=""
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393791    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393794    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393798    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393803    2581 flags.go:64] FLAG: --eviction-soft=""
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393807    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393810    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393813    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393816    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393819    2581 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393822    2581 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393825    2581 flags.go:64] FLAG: --feature-gates=""
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393831    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393834    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393838    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393841    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393844    2581 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393847    2581 flags.go:64] FLAG: --help="false"
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393850    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-130-245.ec2.internal"
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393854    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393857    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393860    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393863    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 00:33:55.400066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393867    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393869    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393872    2581 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393875    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393878    2581 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393881    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393884    2581 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393887    2581 flags.go:64] FLAG: --kube-reserved=""
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393890    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393893    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393896    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393899    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393902    2581 flags.go:64] FLAG: --lock-file=""
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393906    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393909    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393912    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393917    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393920    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393924    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393927    2581 flags.go:64] FLAG: --logging-format="text"
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393930    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393935    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393938    2581 flags.go:64] FLAG: --manifest-url=""
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393942    2581 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393947    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 00:33:55.400854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393950    2581 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393954    2581 flags.go:64] FLAG: --max-pods="110"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393957    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393961    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393964    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393967    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393970    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393973    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393976    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393983    2581 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393986    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393989    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393992    2581 flags.go:64] FLAG: --pod-cidr=""
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393995    2581 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.393998    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394002    2581 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394005    2581 flags.go:64] FLAG: --port="10250"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394008    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394011    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-031dafe1c982a55ca"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394014    2581 flags.go:64] FLAG: --qos-reserved=""
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394019    2581 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394022    2581 flags.go:64] FLAG: --register-node="true"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394025    2581 flags.go:64] FLAG: --register-with-taints=""
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394029    2581 flags.go:64] FLAG: --registry-burst="10"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394031    2581 flags.go:64] FLAG: --registry-qps="5"
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394034    2581 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 00:33:55.401493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394037    2581 flags.go:64] FLAG: --reserved-memory=""
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394041    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394053    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394058    2581 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394061    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394065    2581 flags.go:64] FLAG: --runonce="false"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394068    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394071    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394075    2581 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394078    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394080    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394084    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394087    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394090    2581 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394093    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394096    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394099    2581 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394102    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394105    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394108    2581 flags.go:64] FLAG: --system-cgroups=""
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394111    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394116    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394119    2581 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394122    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394125    2581 flags.go:64] FLAG: --tls-min-version=""
Oct 30 00:33:55.402244 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394128    2581 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394133    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394136    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394139    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394142    2581 flags.go:64] FLAG: --v="2"
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394147    2581 flags.go:64] FLAG: --version="false"
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394151    2581 flags.go:64] FLAG: --vmodule=""
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394155    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394158    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394289    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394293    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394296    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394298    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394302    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394305    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394307    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394310    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394313    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394315    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394318    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394321    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:33:55.402915 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394323    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394326    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394328    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394331    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394333    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394336    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394338    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394341    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394343    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394346    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394349    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394351    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394356    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394358    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394361    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394364    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394367    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394369    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394372    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394374    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:33:55.403430 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394378    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394381    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394383    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394386    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394388    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394391    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394394    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394397    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394399    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394401    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394405    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394409    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394411    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394414    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394417    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394420    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394422    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394425    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394427    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:33:55.403914 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394429    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394432    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394434    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394437    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394440    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394444    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394448    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394451    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394455    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394458    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394461    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394465    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394467    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394471    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394474    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394476    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394479    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394482    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394484    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:33:55.404354 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394487    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394490    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394492    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394495    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394497    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394500    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394502    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394505    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394507    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394510    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394512    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394515    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394517    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394520    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394523    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394525    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394527    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394530    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394534    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394537    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:33:55.405459 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394539    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:33:55.405946 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394542    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:33:55.405946 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394545    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:33:55.405946 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394547    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:33:55.405946 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394550    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:33:55.405946 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394553    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:33:55.405946 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394556    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:33:55.405946 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394559    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:33:55.405946 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394561    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:33:55.405946 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394564    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:33:55.405946 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394567    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:33:55.405946 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394569    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:33:55.405946 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394572    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:33:55.405946 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394575    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:33:55.405946 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394577    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:33:55.405946 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394580    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:33:55.405946 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.394605    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394659    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394664    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394668    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394671    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394673    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394676    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394678    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394681    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394684    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394687    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394689    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394692    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394694    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394697    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394701    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394718    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394723    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394727    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394731    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:33:55.406319 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394734    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394736    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394740    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394743    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394746    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394748    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394751    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394753    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394756    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394759    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394762    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394765    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394768    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394770    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394773    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394776    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394778    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394780    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394783    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:33:55.406846 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394785    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394788    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394790    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394793    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394795    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394798    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394800    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394803    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394807    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394809    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394812    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394814    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394817    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394819    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394822    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394826    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394829    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394831    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394834    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394836    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:33:55.407285 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394839    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394841    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394844    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394847    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394850    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394852    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394855    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394857    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394860    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394863    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394865    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394867    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394870    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394873    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394877    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394879    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394882    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394884    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394887    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394889    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:33:55.407763 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394893    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394896    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394898    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394901    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394903    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394906    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394908    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394912    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394916    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394920    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394923    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394926    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394929    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394931    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394934    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394937    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394939    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394942    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394945    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:33:55.408325 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394948    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394950    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394953    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394955    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394959    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394963    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394968    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394970    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.394973    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.403175    2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.403191    2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.404015    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.404022    2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404082    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404087    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404091    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404094    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404097    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404099    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:33:55.408796 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404102    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404105    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404107    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404110    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404113    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404115    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404118    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404121    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404124    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404127    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404130    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404132    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404135    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404137    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404140    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404143    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404145    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404148    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404151    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404153    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:33:55.409275 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404156    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404159    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404162    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404166    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404168    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404171    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404174    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404176    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404179    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404182    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404184    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404187    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404190    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404192    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404196    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404200    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404203    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404206    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404208    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:33:55.409748 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404211    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404214    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404216    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404219    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404221    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404224    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404227    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404230    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404233    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404235    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404238    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404240    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404243    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404245    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404248    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404251    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404254    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404257    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404259    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404262    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:33:55.410200 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404265    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404268    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404271    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404273    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404276    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404278    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404282    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404284    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404287    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404289    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404292    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404295    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404297    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404300    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404303    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404306    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404308    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404311    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404313    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:33:55.410662 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404316    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404318    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404321    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404324    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404326    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404329    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404331    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404334    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404337    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404340    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404343    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404346    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404348    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404351    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404354    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404357    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404359    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404362    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404366    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404370    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:33:55.411163 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404373    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:33:55.411631 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404376    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:33:55.411631 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.404403    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:33:55.411631 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404542    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:33:55.411631 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404546    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:33:55.411631 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404549    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:33:55.411631 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404552    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:33:55.411631 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404554    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:33:55.411631 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404557    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:33:55.411631 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404560    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:33:55.411631 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404562    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:33:55.411631 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404565    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:33:55.411631 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404568    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:33:55.411631 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404570    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:33:55.411631 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404573    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:33:55.411631 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404576    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:33:55.411631 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404578    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404581    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404583    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404587    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404591    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404594    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404597    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404600    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404602    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404605    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404608    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404610    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404613    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404615    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404618    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404620    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404623    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404625    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404628    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:33:55.412016 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404630    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404633    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404636    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404638    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404641    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404643    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404647    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404649    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404652    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404654    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404657    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404659    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404662    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404664    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404667    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404669    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404672    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404674    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404677    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404680    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:33:55.412482 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404683    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404685    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404688    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404691    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404693    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404696    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404698    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404701    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404716    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404719    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404721    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404724    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404727    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404729    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404732    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404735    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404738    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404742    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404745    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404749    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:33:55.412951 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404752    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404754    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404757    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404760    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404762    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404765    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404767    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404770    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404773    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404775    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404778    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404781    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404784    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404786    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404789    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404792    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404794    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404797    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404800    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:33:55.413403 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404802    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:33:55.413892 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404805    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:33:55.413892 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404807    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:33:55.413892 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404810    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:33:55.413892 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404812    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:33:55.413892 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404815    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:33:55.413892 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404817    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:33:55.413892 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404820    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:33:55.413892 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404822    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:33:55.413892 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404825    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:33:55.413892 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404827    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:33:55.413892 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404830    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:33:55.413892 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404832    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:33:55.413892 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404835    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:33:55.413892 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404838    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:33:55.413892 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.404862    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:33:55.413892 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404894    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404898    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404901    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404904    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404907    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404909    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404912    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404914    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404918    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404922    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404926    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404928    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404931    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404933    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404936    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404939    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404941    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404944    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404946    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:33:55.414295 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404949    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404952    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404954    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404957    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404959    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404962    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404964    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404967    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404969    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404971    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404974    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404977    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404979    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404982    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404985    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404987    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404990    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404993    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404995    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.404997    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:33:55.414744 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405000    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405003    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405005    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405008    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405011    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405013    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405016    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405019    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405021    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405024    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405026    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405029    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405031    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405034    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405036    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405039    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405041    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405044    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405046    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405049    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:33:55.415201 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405051    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405056    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405059    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405062    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405067    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405071    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405073    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405076    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405078    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405081    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405083    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405086    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405089    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405091    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405094    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405096    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405099    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405101    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405104    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:33:55.415692 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405106    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405109    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405111    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405114    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405117    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405119    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405122    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405125    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405128    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405130    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405133    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405135    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405138    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405140    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405143    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405146    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405148    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405152    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405156    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:33:55.416142 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405159    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:33:55.416595 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405162    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:33:55.416595 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405165    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:33:55.416595 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405167    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:33:55.416595 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405170    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:33:55.416595 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405172    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:33:55.416595 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405175    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:33:55.416595 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405178    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:33:55.416595 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:55.405180    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:33:55.416595 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.405235    2581 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 00:33:55.416595 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.408117    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 00:33:55.416595 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.408925    2581 server.go:1014] "Starting client certificate rotation"
Oct 30 00:33:55.416595 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.409021    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:33:55.416595 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.409061    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:33:55.416595 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.410714    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:33:55.416595 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.411989    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:33:55.423894 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.423878    2581 log.go:25] "Validated CRI v1 runtime API"
Oct 30 00:33:55.427513 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.427498    2581 log.go:25] "Validated CRI v1 image API"
Oct 30 00:33:55.428297 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.428271    2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 00:33:55.430851 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.430827    2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b9afe5ff-e2a3-4f26-9e90-fc8aa7e6a9c6:/dev/nvme0n1p3 f64bb1db-327b-4db0-b50d-925030ff8ef0:/dev/nvme0n1p4]
Oct 30 00:33:55.430912 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.430849    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 00:33:55.437311 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.436954    2581 manager.go:217] Machine: {Timestamp:2026-10-30 00:33:55.434760511 +0000 UTC m=+0.307516725 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098880 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ecc5a2129e52e70e688440f2119ed SystemUUID:ec2ecc5a-2129-e52e-70e6-88440f2119ed BootID:e1d50569-5e03-478a-80bc-e3f387eb528e Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:0e:2b:a5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:0e:2b:a5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:be:42:0f:3f:1b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 00:33:55.437910 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.437897    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 00:33:55.437999 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.437987    2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 00:33:55.440921 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.440890    2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 00:33:55.441075 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.440923    2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-245.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 00:33:55.441163 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.441080    2581 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 00:33:55.441163 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.441092    2581 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 00:33:55.441163 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.441110    2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:33:55.441926 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.441915    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:33:55.442741 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.442730    2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 00:33:55.446463 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.446448    2581 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 00:33:55.446593 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.446582    2581 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 00:33:55.447308 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.447298    2581 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 00:33:55.447373 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.447316    2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 00:33:55.447373 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.447329    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 00:33:55.447373 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.447342    2581 kubelet.go:405] "Adding apiserver pod source"
Oct 30 00:33:55.447373 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.447353    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 00:33:55.447674 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.447661    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:33:55.449832 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.449812    2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 00:33:55.455080 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.455067    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:33:55.455824 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.455812    2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 00:33:55.455865 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.455842    2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 00:33:55.455985 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.455976    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 00:33:55.456022 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.455992    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 00:33:55.456022 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.455998    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 00:33:55.456022 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.456003    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 00:33:55.456022 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.456009    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 00:33:55.456022 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.456014    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 00:33:55.456570 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.456562    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 00:33:55.456600 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.456571    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 00:33:55.456600 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.456578    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 00:33:55.456600 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.456584    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 00:33:55.456600 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.456592    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 00:33:55.456600 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.456601    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 00:33:55.457327 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.457319    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 00:33:55.457357 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.457328    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 00:33:55.460941 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.460930    2581 server.go:1263] "Started kubelet"
Oct 30 00:33:55.461021 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.460980    2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 00:33:55.461090 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.461036    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 00:33:55.461134 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.461120    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 00:33:55.461628 ip-10-0-130-245 systemd[1]: Started Kubernetes Kubelet.
Oct 30 00:33:55.462163 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.462146    2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 00:33:55.463001 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.462986    2581 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 00:33:55.468498 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.468482    2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 00:33:55.469464 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.469452    2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 00:33:55.469522 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.469469    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 00:33:55.469588 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.469526    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found"
Oct 30 00:33:55.471213 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.470037    2581 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 00:33:55.471213 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.470076    2581 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 00:33:55.471213 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.470038    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 00:33:55.471213 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.470214    2581 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 00:33:55.471213 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.470222    2581 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 00:33:55.471213 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.470351    2581 factory.go:55] Registering systemd factory
Oct 30 00:33:55.471213 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.470364    2581 factory.go:223] Registration of the systemd container factory successfully
Oct 30 00:33:55.471213 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.470548    2581 factory.go:153] Registering CRI-O factory
Oct 30 00:33:55.471213 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.470560    2581 factory.go:223] Registration of the crio container factory successfully
Oct 30 00:33:55.471213 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.470649    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 00:33:55.471213 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.470665    2581 factory.go:103] Registering Raw factory
Oct 30 00:33:55.471213 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.470679    2581 manager.go:1199] Started watching for new ooms in manager
Oct 30 00:33:55.471758 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.471634    2581 manager.go:319] Starting recovery of all containers
Oct 30 00:33:55.483460 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.483441    2581 manager.go:324] Recovery completed
Oct 30 00:33:55.488978 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.488962    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:33:55.495883 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.495860    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:33:55.495955 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.495897    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:33:55.495955 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.495911    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:33:55.496433 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.496419    2581 cpu_manager.go:226] "Starting" policy="none"
Oct 30 00:33:55.496500 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.496429    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:33:55.496500 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.496432    2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 00:33:55.496500 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.496489    2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 00:33:55.497151 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.497135    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:33:55.497196 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.497164    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:33:55.497196 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.497175    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:33:55.498825 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.498800    2581 policy_none.go:50] "Start"
Oct 30 00:33:55.498825 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.498827    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 00:33:55.498963 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.498841    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 00:33:55.500859 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.500844    2581 policy_none.go:44] "Start"
Oct 30 00:33:55.502274 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.502253    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-245.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 00:33:55.502274 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.502251    2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-245.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 00:33:55.502492 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.502470    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 00:33:55.503175 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.502226    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-245.ec2.internal.18e3276f441898d9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-245.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-245.ec2.internal,},FirstTimestamp:2026-10-30 00:33:55.460905177 +0000 UTC m=+0.333661390,LastTimestamp:2026-10-30 00:33:55.460905177 +0000 UTC m=+0.333661390,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-245.ec2.internal,}"
Oct 30 00:33:55.513412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.513385    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-srnqs"
Oct 30 00:33:55.520203 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.520182    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-245.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 00:33:55.520277 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.520201    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 00:33:55.520277 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.520194    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-245.ec2.internal.18e3276f462e4f2e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-245.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-245.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-245.ec2.internal,},FirstTimestamp:2026-10-30 00:33:55.495882542 +0000 UTC m=+0.368638762,LastTimestamp:2026-10-30 00:33:55.495882542 +0000 UTC m=+0.368638762,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-245.ec2.internal,}"
Oct 30 00:33:55.520521 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.520500    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 00:33:55.525206 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.525191    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-srnqs"
Oct 30 00:33:55.535260 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.535233    2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 00:33:55.537607 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.537595    2581 manager.go:341] "Starting Device Plugin manager"
Oct 30 00:33:55.538226 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.538211    2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 00:33:55.538226 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.538227    2581 server.go:87] "Starting device plugin registration server"
Oct 30 00:33:55.538420 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.538410    2581 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 00:33:55.538470 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.538423    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 00:33:55.538587 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.538574    2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 00:33:55.538666 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.538647    2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 00:33:55.538666 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.538656    2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 00:33:55.539179 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.539164    2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 00:33:55.539232 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.539206    2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-245.ec2.internal\" not found"
Oct 30 00:33:55.603427 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.603392    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 00:33:55.605543 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.604626    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 00:33:55.605543 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.604648    2581 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 00:33:55.605543 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.604668    2581 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 00:33:55.605543 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.604720    2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 00:33:55.617909 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.617893    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:33:55.638773 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.638732    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:33:55.639531 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.639512    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:33:55.639590 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.639542    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:33:55.639590 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.639558    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:33:55.639590 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.639582    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-245.ec2.internal"
Oct 30 00:33:55.657783 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.657768    2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-245.ec2.internal"
Oct 30 00:33:55.657867 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.657788    2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-245.ec2.internal\": node \"ip-10-0-130-245.ec2.internal\" not found"
Oct 30 00:33:55.695500 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.695480    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found"
Oct 30 00:33:55.705099 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.705082    2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"]
Oct 30 00:33:55.705146 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.705127    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:33:55.705847 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.705834    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:33:55.705918 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.705857    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:33:55.705918 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.705866    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:33:55.708116 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.708104    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:33:55.708262 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.708249    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal"
Oct 30 00:33:55.708297 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.708276    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:33:55.708857 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.708835    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:33:55.708920 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.708872    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:33:55.708920 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.708881    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:33:55.708981 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.708844    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:33:55.708981 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.708950    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:33:55.708981 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.708964    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:33:55.711019 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.711005    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:33:55.711089 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.711030    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:33:55.711718 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.711690    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:33:55.711798 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.711729    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:33:55.711798 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.711744    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:33:55.742306 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.742279    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-245.ec2.internal\" not found" node="ip-10-0-130-245.ec2.internal"
Oct 30 00:33:55.748982 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.748967    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-245.ec2.internal\" not found" node="ip-10-0-130-245.ec2.internal"
Oct 30 00:33:55.771992 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.771975    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/553cfdac89599e545ab5a9539f3d3b39-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" (UID: \"553cfdac89599e545ab5a9539f3d3b39\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:33:55.772061 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.772000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f8c3a9edbd8cbf03175beab8c4db7242-config\") pod \"kube-apiserver-proxy-ip-10-0-130-245.ec2.internal\" (UID: \"f8c3a9edbd8cbf03175beab8c4db7242\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal"
Oct 30 00:33:55.772061 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.772016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/553cfdac89599e545ab5a9539f3d3b39-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" (UID: \"553cfdac89599e545ab5a9539f3d3b39\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:33:55.795757 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.795737    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found"
Oct 30 00:33:55.873132 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.873110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f8c3a9edbd8cbf03175beab8c4db7242-config\") pod \"kube-apiserver-proxy-ip-10-0-130-245.ec2.internal\" (UID: \"f8c3a9edbd8cbf03175beab8c4db7242\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal"
Oct 30 00:33:55.873217 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.873138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/553cfdac89599e545ab5a9539f3d3b39-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" (UID: \"553cfdac89599e545ab5a9539f3d3b39\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:33:55.873217 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.873162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/553cfdac89599e545ab5a9539f3d3b39-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" (UID: \"553cfdac89599e545ab5a9539f3d3b39\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:33:55.873300 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.873225    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f8c3a9edbd8cbf03175beab8c4db7242-config\") pod \"kube-apiserver-proxy-ip-10-0-130-245.ec2.internal\" (UID: \"f8c3a9edbd8cbf03175beab8c4db7242\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal"
Oct 30 00:33:55.873300 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.873233    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/553cfdac89599e545ab5a9539f3d3b39-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" (UID: \"553cfdac89599e545ab5a9539f3d3b39\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:33:55.873300 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:55.873279    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/553cfdac89599e545ab5a9539f3d3b39-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" (UID: \"553cfdac89599e545ab5a9539f3d3b39\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:33:55.895933 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.895887    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found"
Oct 30 00:33:55.996105 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:55.996079    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found"
Oct 30 00:33:56.046291 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.046270    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal"
Oct 30 00:33:56.053006 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.052991    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:33:56.097123 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:56.096994    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found"
Oct 30 00:33:56.197521 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:56.197477    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found"
Oct 30 00:33:56.297956 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:56.297932    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found"
Oct 30 00:33:56.345068 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.345052    2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:33:56.370161 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.370142    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal"
Oct 30 00:33:56.388750 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.388735    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:33:56.390812 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.390799    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:33:56.408817 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.408794    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 00:33:56.408954 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:56.408931    2581 kubelet.go:3370] "Failed creating a mirror pod" err="Post \"https://a8954c67cbcde4fa493ff97799e1acda-b3794413c9eb4e0c.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.130.245:38358->3.146.242.70:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:33:56.408954 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.408946    2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 00:33:56.409036 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.408953    2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 00:33:56.448438 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.448379    2581 apiserver.go:52] "Watching apiserver"
Oct 30 00:33:56.469767 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.469748    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 00:33:56.529859 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.529822    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:28:56 +0000 UTC" deadline="2028-03-25 00:38:04.360241857 +0000 UTC"
Oct 30 00:33:56.529859 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.529857    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12288h4m7.830387654s"
Oct 30 00:33:56.566167 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.566145    2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 00:33:56.566496 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.566478    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-whx68","openshift-ovn-kubernetes/ovnkube-node-vcqkc","kube-system/global-pull-secret-syncer-7vsmg","openshift-cluster-node-tuning-operator/tuned-9h5hq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v","openshift-multus/network-metrics-daemon-wb6st","kube-system/konnectivity-agent-v6rdh","openshift-dns/node-resolver-b2bnk","openshift-image-registry/node-ca-d9lhd","openshift-network-diagnostics/network-check-target-62gpq","openshift-multus/multus-42gzj","openshift-multus/multus-additional-cni-plugins-gmgzr"]
Oct 30 00:33:56.568965 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.568946    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.572018 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.571996    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d9lhd"
Oct 30 00:33:56.574948 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.574928    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:33:56.575047 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:56.575007    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:33:56.577103 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.577083    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.577211 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.577088    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.577267 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.577197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-socket-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.577267 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.577237    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhvbj\" (UniqueName: \"kubernetes.io/projected/733527bc-83b7-423c-a79e-1de279f5a607-kube-api-access-jhvbj\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.577333 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.577266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5975a832-beb3-4e76-98ad-aef36a2ccce2-host\") pod \"node-ca-d9lhd\" (UID: \"5975a832-beb3-4e76-98ad-aef36a2ccce2\") " pod="openshift-image-registry/node-ca-d9lhd"
Oct 30 00:33:56.577333 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.577292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5975a832-beb3-4e76-98ad-aef36a2ccce2-serviceca\") pod \"node-ca-d9lhd\" (UID: \"5975a832-beb3-4e76-98ad-aef36a2ccce2\") " pod="openshift-image-registry/node-ca-d9lhd"
Oct 30 00:33:56.577434 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.577349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-registration-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.577434 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.577401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-device-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.577495 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.577463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-etc-selinux\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.577534 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.577499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-sys-fs\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.577534 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.577518    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnd4w\" (UniqueName: \"kubernetes.io/projected/5975a832-beb3-4e76-98ad-aef36a2ccce2-kube-api-access-cnd4w\") pod \"node-ca-d9lhd\" (UID: \"5975a832-beb3-4e76-98ad-aef36a2ccce2\") " pod="openshift-image-registry/node-ca-d9lhd"
Oct 30 00:33:56.579420 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.579402    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.581735 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.581706    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:33:56.581845 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:56.581769    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:33:56.584893 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.584873    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v6rdh"
Oct 30 00:33:56.586302 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.586278    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 00:33:56.586379 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.586342    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 00:33:56.586422 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.586386    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ltmbf\""
Oct 30 00:33:56.586511 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.586496    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 00:33:56.587196 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.587181    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.588700 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.588685    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 00:33:56.591236 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.589448    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 00:33:56.591236 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.589556    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 00:33:56.591236 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.589590    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-s9968\""
Oct 30 00:33:56.591733 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.591705    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.594099 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.594081    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b2bnk"
Oct 30 00:33:56.596542 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.596524    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-whx68"
Oct 30 00:33:56.599619 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.599596    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:33:56.599700 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:56.599669    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:33:56.635700 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:56.635675    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c3a9edbd8cbf03175beab8c4db7242.slice/crio-ab0c9e3bbd550a22c57907f05402ac9314905c0c759dec1a19a3abf77b32903e WatchSource:0}: Error finding container ab0c9e3bbd550a22c57907f05402ac9314905c0c759dec1a19a3abf77b32903e: Status 404 returned error can't find the container with id ab0c9e3bbd550a22c57907f05402ac9314905c0c759dec1a19a3abf77b32903e
Oct 30 00:33:56.643087 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.643070    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:33:56.646280 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:56.646259    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553cfdac89599e545ab5a9539f3d3b39.slice/crio-edaebab344eb8b504f4cbb62b8b245ce7e6a502dfbe2011df04d7030ddd7e6e8 WatchSource:0}: Error finding container edaebab344eb8b504f4cbb62b8b245ce7e6a502dfbe2011df04d7030ddd7e6e8: Status 404 returned error can't find the container with id edaebab344eb8b504f4cbb62b8b245ce7e6a502dfbe2011df04d7030ddd7e6e8
Oct 30 00:33:56.650224 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.650204    2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:33:56.671864 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.671831    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 00:33:56.677702 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.677651    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-sysctl-conf\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.677702 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.677690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32086945-d23d-49bb-a876-fab4cef52ec6-host-slash\") pod \"iptables-alerter-whx68\" (UID: \"32086945-d23d-49bb-a876-fab4cef52ec6\") " pod="openshift-network-operator/iptables-alerter-whx68"
Oct 30 00:33:56.677862 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.677719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbhmk\" (UniqueName: \"kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk\") pod \"network-check-target-62gpq\" (UID: \"c9803898-12ec-4a07-9a8f-1eef3f89235b\") " pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:33:56.677862 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.677764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b7f00b4a-044e-49ab-8f43-4351590692e8-kubelet-config\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:33:56.677862 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.677793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b7f00b4a-044e-49ab-8f43-4351590692e8-dbus\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:33:56.677862 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.677854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-node-log\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.678073 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.677888    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs\") pod \"network-metrics-daemon-wb6st\" (UID: \"3a145ece-ad48-43c2-a259-b9590802cfe8\") " pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:33:56.678073 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.677908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-var-lib-cni-bin\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.678073 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.677962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-registration-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.678073 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.677994    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-systemd\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.678073 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678049    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-registration-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.678340 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678073    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/070879ac-09e8-4fdb-b55f-e52a9998552f-ovnkube-config\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.678340 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678116    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/070879ac-09e8-4fdb-b55f-e52a9998552f-ovnkube-script-lib\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.678340 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678155    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-257nk\" (UniqueName: \"kubernetes.io/projected/070879ac-09e8-4fdb-b55f-e52a9998552f-kube-api-access-257nk\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.678340 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmqr7\" (UniqueName: \"kubernetes.io/projected/5ac72880-98ab-4093-b7aa-903a37c8113e-kube-api-access-bmqr7\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.678340 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678229    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5975a832-beb3-4e76-98ad-aef36a2ccce2-serviceca\") pod \"node-ca-d9lhd\" (UID: \"5975a832-beb3-4e76-98ad-aef36a2ccce2\") " pod="openshift-image-registry/node-ca-d9lhd"
Oct 30 00:33:56.678340 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678284    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-tmp\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.678340 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d996f136-ba95-4636-a870-94520d3eff0f-system-cni-dir\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.678688 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678365    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-system-cni-dir\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.678688 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678415    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ac72880-98ab-4093-b7aa-903a37c8113e-cni-binary-copy\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.678688 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-etc-selinux\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.678688 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-lib-modules\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.678688 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d996f136-ba95-4636-a870-94520d3eff0f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.678688 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678567    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d996f136-ba95-4636-a870-94520d3eff0f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.678688 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678606    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-etc-selinux\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.678688 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678631    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-etc-openvswitch\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.678688 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nbn8\" (UniqueName: \"kubernetes.io/projected/3a145ece-ad48-43c2-a259-b9590802cfe8-kube-api-access-8nbn8\") pod \"network-metrics-daemon-wb6st\" (UID: \"3a145ece-ad48-43c2-a259-b9590802cfe8\") " pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:33:56.678688 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678687    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-etc-kubernetes\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.679241 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-run\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.679241 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678718    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5975a832-beb3-4e76-98ad-aef36a2ccce2-serviceca\") pod \"node-ca-d9lhd\" (UID: \"5975a832-beb3-4e76-98ad-aef36a2ccce2\") " pod="openshift-image-registry/node-ca-d9lhd"
Oct 30 00:33:56.679241 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-socket-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.679241 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32086945-d23d-49bb-a876-fab4cef52ec6-iptables-alerter-script\") pod \"iptables-alerter-whx68\" (UID: \"32086945-d23d-49bb-a876-fab4cef52ec6\") " pod="openshift-network-operator/iptables-alerter-whx68"
Oct 30 00:33:56.679241 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678855    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-socket-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.679241 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678884    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-var-lib-kubelet\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.679241 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678911    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d996f136-ba95-4636-a870-94520d3eff0f-cnibin\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.679241 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678929    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d996f136-ba95-4636-a870-94520d3eff0f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.679241 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.678955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-var-lib-openvswitch\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.679241 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-sys-fs\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.679241 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbtjm\" (UniqueName: \"kubernetes.io/projected/d996f136-ba95-4636-a870-94520d3eff0f-kube-api-access-vbtjm\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.679241 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679049    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-cni-bin\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.679241 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679065    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-cni-netd\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.679241 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47d34735-e4b1-4073-b6de-98a9a720432b-konnectivity-ca\") pod \"konnectivity-agent-v6rdh\" (UID: \"47d34735-e4b1-4073-b6de-98a9a720432b\") " pod="kube-system/konnectivity-agent-v6rdh"
Oct 30 00:33:56.679241 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679141    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-sys-fs\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.679241 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679183    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-run-k8s-cni-cncf-io\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.679989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-hostroot\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.679989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679230    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-conf-dir\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.679989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-run-openvswitch\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.679989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-var-lib-cni-multus\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.679989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679350    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7-hosts-file\") pod \"node-resolver-b2bnk\" (UID: \"b45c13d7-29c0-4054-9b7d-fd33ecec9cf7\") " pod="openshift-dns/node-resolver-b2bnk"
Oct 30 00:33:56.679989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7-tmp-dir\") pod \"node-resolver-b2bnk\" (UID: \"b45c13d7-29c0-4054-9b7d-fd33ecec9cf7\") " pod="openshift-dns/node-resolver-b2bnk"
Oct 30 00:33:56.679989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-log-socket\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.679989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-daemon-config\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.679989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679525    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-sysconfig\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.679989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhvbj\" (UniqueName: \"kubernetes.io/projected/733527bc-83b7-423c-a79e-1de279f5a607-kube-api-access-jhvbj\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.679989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679745    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9rmw\" (UniqueName: \"kubernetes.io/projected/32086945-d23d-49bb-a876-fab4cef52ec6-kube-api-access-d9rmw\") pod \"iptables-alerter-whx68\" (UID: \"32086945-d23d-49bb-a876-fab4cef52ec6\") " pod="openshift-network-operator/iptables-alerter-whx68"
Oct 30 00:33:56.679989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-systemd-units\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.679989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-cnibin\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.679989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679869    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-run-netns\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.679989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnd4w\" (UniqueName: \"kubernetes.io/projected/5975a832-beb3-4e76-98ad-aef36a2ccce2-kube-api-access-cnd4w\") pod \"node-ca-d9lhd\" (UID: \"5975a832-beb3-4e76-98ad-aef36a2ccce2\") " pod="openshift-image-registry/node-ca-d9lhd"
Oct 30 00:33:56.679989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679943    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-run-ovn\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.679989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.679986    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-cni-dir\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.680704 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5975a832-beb3-4e76-98ad-aef36a2ccce2-host\") pod \"node-ca-d9lhd\" (UID: \"5975a832-beb3-4e76-98ad-aef36a2ccce2\") " pod="openshift-image-registry/node-ca-d9lhd"
Oct 30 00:33:56.680704 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-sys\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.680704 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-run-netns\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.680704 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680103    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5975a832-beb3-4e76-98ad-aef36a2ccce2-host\") pod \"node-ca-d9lhd\" (UID: \"5975a832-beb3-4e76-98ad-aef36a2ccce2\") " pod="openshift-image-registry/node-ca-d9lhd"
Oct 30 00:33:56.680704 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-device-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.680704 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-sysctl-d\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.680704 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680197    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-device-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.680704 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-slash\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.680704 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680252    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.680704 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680270    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-var-lib-kubelet\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.680704 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.680704 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680358    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:33:56.680704 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680393    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.680704 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680399    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-run-multus-certs\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.680704 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-modprobe-d\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.680704 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680474    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-tuned\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.680704 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-kubelet\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.681314 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-run-systemd\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.681314 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.681314 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/070879ac-09e8-4fdb-b55f-e52a9998552f-ovn-node-metrics-cert\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.681314 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-socket-dir-parent\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.681314 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-kubernetes\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.681314 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f24vm\" (UniqueName: \"kubernetes.io/projected/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-kube-api-access-f24vm\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.681314 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d996f136-ba95-4636-a870-94520d3eff0f-cni-binary-copy\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.681314 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680799    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/070879ac-09e8-4fdb-b55f-e52a9998552f-env-overrides\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.681314 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4tjg\" (UniqueName: \"kubernetes.io/projected/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7-kube-api-access-f4tjg\") pod \"node-resolver-b2bnk\" (UID: \"b45c13d7-29c0-4054-9b7d-fd33ecec9cf7\") " pod="openshift-dns/node-resolver-b2bnk"
Oct 30 00:33:56.681314 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-host\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.681314 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.680868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47d34735-e4b1-4073-b6de-98a9a720432b-agent-certs\") pod \"konnectivity-agent-v6rdh\" (UID: \"47d34735-e4b1-4073-b6de-98a9a720432b\") " pod="kube-system/konnectivity-agent-v6rdh"
Oct 30 00:33:56.691653 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.691632    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 00:33:56.691748 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.691688    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rpn9f"
Oct 30 00:33:56.695774 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.695735    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 00:33:56.695774 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.695763    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 00:33:56.696030 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.696009    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qff2g\""
Oct 30 00:33:56.696095 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.696070    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 00:33:56.696095 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.696077    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:33:56.696209 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.696196    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 00:33:56.696261 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.696246    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 00:33:56.696315 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.696293    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b4svw\""
Oct 30 00:33:56.696315 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.696298    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-x59g5\""
Oct 30 00:33:56.696444 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.696286    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 00:33:56.696444 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.696343    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 00:33:56.696548 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.696492    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:33:56.696548 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.696503    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 00:33:56.696548 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.696520    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 00:33:56.696653 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.696552    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 00:33:56.708922 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.706900    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 00:33:56.711074 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.711054    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 00:33:56.711074 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.711070    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 00:33:56.711228 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.711085    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tcw96\""
Oct 30 00:33:56.711228 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.711070    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 00:33:56.711228 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.711122    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 00:33:56.711228 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.711134    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 00:33:56.711228 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.711122    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 00:33:56.711655 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.711474    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jk5pr\""
Oct 30 00:33:56.711655 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.711492    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q2pzq\""
Oct 30 00:33:56.711655 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.711481    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 00:33:56.711655 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.711509    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 00:33:56.711655 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.711520    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qpxdw\""
Oct 30 00:33:56.712663 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.712636    2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 00:33:56.716316 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.716298    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnd4w\" (UniqueName: \"kubernetes.io/projected/5975a832-beb3-4e76-98ad-aef36a2ccce2-kube-api-access-cnd4w\") pod \"node-ca-d9lhd\" (UID: \"5975a832-beb3-4e76-98ad-aef36a2ccce2\") " pod="openshift-image-registry/node-ca-d9lhd"
Oct 30 00:33:56.716316 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.716309    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhvbj\" (UniqueName: \"kubernetes.io/projected/733527bc-83b7-423c-a79e-1de279f5a607-kube-api-access-jhvbj\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.781878 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.781855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f24vm\" (UniqueName: \"kubernetes.io/projected/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-kube-api-access-f24vm\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.781975 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.781883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d996f136-ba95-4636-a870-94520d3eff0f-cni-binary-copy\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.781975 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.781900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/070879ac-09e8-4fdb-b55f-e52a9998552f-env-overrides\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.781975 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.781924    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4tjg\" (UniqueName: \"kubernetes.io/projected/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7-kube-api-access-f4tjg\") pod \"node-resolver-b2bnk\" (UID: \"b45c13d7-29c0-4054-9b7d-fd33ecec9cf7\") " pod="openshift-dns/node-resolver-b2bnk"
Oct 30 00:33:56.781975 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.781947    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-host\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.781975 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.781971    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47d34735-e4b1-4073-b6de-98a9a720432b-agent-certs\") pod \"konnectivity-agent-v6rdh\" (UID: \"47d34735-e4b1-4073-b6de-98a9a720432b\") " pod="kube-system/konnectivity-agent-v6rdh"
Oct 30 00:33:56.782235 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.781996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-sysctl-conf\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.782235 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782022    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32086945-d23d-49bb-a876-fab4cef52ec6-host-slash\") pod \"iptables-alerter-whx68\" (UID: \"32086945-d23d-49bb-a876-fab4cef52ec6\") " pod="openshift-network-operator/iptables-alerter-whx68"
Oct 30 00:33:56.782235 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhmk\" (UniqueName: \"kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk\") pod \"network-check-target-62gpq\" (UID: \"c9803898-12ec-4a07-9a8f-1eef3f89235b\") " pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:33:56.782235 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782065    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-host\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.782235 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b7f00b4a-044e-49ab-8f43-4351590692e8-kubelet-config\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:33:56.782235 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782130    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b7f00b4a-044e-49ab-8f43-4351590692e8-kubelet-config\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:33:56.782235 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b7f00b4a-044e-49ab-8f43-4351590692e8-dbus\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:33:56.782235 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-node-log\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.782235 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782192    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-sysctl-conf\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.782235 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782213    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs\") pod \"network-metrics-daemon-wb6st\" (UID: \"3a145ece-ad48-43c2-a259-b9590802cfe8\") " pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:33:56.782235 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782184    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32086945-d23d-49bb-a876-fab4cef52ec6-host-slash\") pod \"iptables-alerter-whx68\" (UID: \"32086945-d23d-49bb-a876-fab4cef52ec6\") " pod="openshift-network-operator/iptables-alerter-whx68"
Oct 30 00:33:56.782235 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-var-lib-cni-bin\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.782797 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782251    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-node-log\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.782797 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-systemd\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.782797 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782305    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-var-lib-cni-bin\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.782797 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/070879ac-09e8-4fdb-b55f-e52a9998552f-ovnkube-config\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.782797 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:56.782312    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:33:56.782797 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782325    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b7f00b4a-044e-49ab-8f43-4351590692e8-dbus\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:33:56.782797 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/070879ac-09e8-4fdb-b55f-e52a9998552f-ovnkube-script-lib\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.782797 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782355    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-systemd\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.782797 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:56.782401    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs podName:3a145ece-ad48-43c2-a259-b9590802cfe8 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:57.282376221 +0000 UTC m=+2.155132446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs") pod "network-metrics-daemon-wb6st" (UID: "3a145ece-ad48-43c2-a259-b9590802cfe8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:33:56.782797 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-257nk\" (UniqueName: \"kubernetes.io/projected/070879ac-09e8-4fdb-b55f-e52a9998552f-kube-api-access-257nk\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.782797 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782438    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/070879ac-09e8-4fdb-b55f-e52a9998552f-env-overrides\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.782797 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782451    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmqr7\" (UniqueName: \"kubernetes.io/projected/5ac72880-98ab-4093-b7aa-903a37c8113e-kube-api-access-bmqr7\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.782797 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-tmp\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.782797 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782510    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d996f136-ba95-4636-a870-94520d3eff0f-cni-binary-copy\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.782797 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d996f136-ba95-4636-a870-94520d3eff0f-system-cni-dir\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.782797 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782546    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d996f136-ba95-4636-a870-94520d3eff0f-system-cni-dir\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.782797 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782556    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-system-cni-dir\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.783673 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ac72880-98ab-4093-b7aa-903a37c8113e-cni-binary-copy\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.783673 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-lib-modules\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.783673 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d996f136-ba95-4636-a870-94520d3eff0f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.783673 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d996f136-ba95-4636-a870-94520d3eff0f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.783673 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782657    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-system-cni-dir\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.783673 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-etc-openvswitch\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.783673 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nbn8\" (UniqueName: \"kubernetes.io/projected/3a145ece-ad48-43c2-a259-b9590802cfe8-kube-api-access-8nbn8\") pod \"network-metrics-daemon-wb6st\" (UID: \"3a145ece-ad48-43c2-a259-b9590802cfe8\") " pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:33:56.783673 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-etc-kubernetes\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.783673 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-run\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.783673 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32086945-d23d-49bb-a876-fab4cef52ec6-iptables-alerter-script\") pod \"iptables-alerter-whx68\" (UID: \"32086945-d23d-49bb-a876-fab4cef52ec6\") " pod="openshift-network-operator/iptables-alerter-whx68"
Oct 30 00:33:56.783673 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782808    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-lib-modules\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.783673 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782823    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-var-lib-kubelet\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.783673 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d996f136-ba95-4636-a870-94520d3eff0f-cnibin\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.783673 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782861    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-etc-kubernetes\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.783673 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d996f136-ba95-4636-a870-94520d3eff0f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.783673 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-var-lib-openvswitch\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.783673 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782906    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-etc-openvswitch\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.784555 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782909    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/070879ac-09e8-4fdb-b55f-e52a9998552f-ovnkube-config\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.784555 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbtjm\" (UniqueName: \"kubernetes.io/projected/d996f136-ba95-4636-a870-94520d3eff0f-kube-api-access-vbtjm\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.784555 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-cni-bin\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.784555 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782979    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-run\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.784555 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.782980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-cni-netd\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.784555 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783000    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/070879ac-09e8-4fdb-b55f-e52a9998552f-ovnkube-script-lib\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.784555 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47d34735-e4b1-4073-b6de-98a9a720432b-konnectivity-ca\") pod \"konnectivity-agent-v6rdh\" (UID: \"47d34735-e4b1-4073-b6de-98a9a720432b\") " pod="kube-system/konnectivity-agent-v6rdh"
Oct 30 00:33:56.784555 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783023    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-cni-netd\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.784555 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-run-k8s-cni-cncf-io\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.784555 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783073    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d996f136-ba95-4636-a870-94520d3eff0f-cnibin\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.784555 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783081    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-hostroot\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.784555 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783119    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-conf-dir\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.784555 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-run-openvswitch\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.784555 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-var-lib-cni-multus\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.784555 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7-hosts-file\") pod \"node-resolver-b2bnk\" (UID: \"b45c13d7-29c0-4054-9b7d-fd33ecec9cf7\") " pod="openshift-dns/node-resolver-b2bnk"
Oct 30 00:33:56.784555 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783202    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d996f136-ba95-4636-a870-94520d3eff0f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.784555 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7-tmp-dir\") pod \"node-resolver-b2bnk\" (UID: \"b45c13d7-29c0-4054-9b7d-fd33ecec9cf7\") " pod="openshift-dns/node-resolver-b2bnk"
Oct 30 00:33:56.784555 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783237    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ac72880-98ab-4093-b7aa-903a37c8113e-cni-binary-copy\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.785397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783237    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d996f136-ba95-4636-a870-94520d3eff0f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.785397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-log-socket\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.785397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783277    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-cni-bin\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.785397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-daemon-config\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.785397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783297    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-log-socket\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.785397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783316    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-sysconfig\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.785397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783339    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-hostroot\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.785397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rmw\" (UniqueName: \"kubernetes.io/projected/32086945-d23d-49bb-a876-fab4cef52ec6-kube-api-access-d9rmw\") pod \"iptables-alerter-whx68\" (UID: \"32086945-d23d-49bb-a876-fab4cef52ec6\") " pod="openshift-network-operator/iptables-alerter-whx68"
Oct 30 00:33:56.785397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-systemd-units\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.785397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783381    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-var-lib-openvswitch\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.785397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-cnibin\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.785397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-run-netns\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.785397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-run-ovn\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.785397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783459    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-conf-dir\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.785397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-cni-dir\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.785397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783500    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-run-openvswitch\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.785397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783505    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-sys\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.785397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783550    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-var-lib-cni-multus\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.785930 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-run-netns\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.785930 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783592    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-run-netns\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.785930 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-sysctl-d\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.785930 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-slash\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.785930 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783644    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-run-k8s-cni-cncf-io\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.785930 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.785930 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783690    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-sysconfig\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.785930 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-var-lib-kubelet\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.785930 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783709    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d996f136-ba95-4636-a870-94520d3eff0f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.785930 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783724    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-var-lib-kubelet\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.785930 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783734    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:33:56.785930 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783743    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32086945-d23d-49bb-a876-fab4cef52ec6-iptables-alerter-script\") pod \"iptables-alerter-whx68\" (UID: \"32086945-d23d-49bb-a876-fab4cef52ec6\") " pod="openshift-network-operator/iptables-alerter-whx68"
Oct 30 00:33:56.785930 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783775    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7-hosts-file\") pod \"node-resolver-b2bnk\" (UID: \"b45c13d7-29c0-4054-9b7d-fd33ecec9cf7\") " pod="openshift-dns/node-resolver-b2bnk"
Oct 30 00:33:56.785930 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783802    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47d34735-e4b1-4073-b6de-98a9a720432b-konnectivity-ca\") pod \"konnectivity-agent-v6rdh\" (UID: \"47d34735-e4b1-4073-b6de-98a9a720432b\") " pod="kube-system/konnectivity-agent-v6rdh"
Oct 30 00:33:56.785930 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783780    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-run-multus-certs\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.785930 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783835    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-sysctl-d\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.785930 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783842    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-run-ovn\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.786412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783841    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-daemon-config\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.786412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783428    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-var-lib-kubelet\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.786412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-modprobe-d\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.786412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783891    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-run-netns\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.786412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-tuned\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.786412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783899    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-systemd-units\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.786412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783926    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-modprobe-d\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.786412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783945    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-cnibin\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.786412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783960    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.786412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.783972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-kubelet\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.786412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.784003    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-kubelet\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.786412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.784005    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-run-systemd\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.786412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.784017    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-cni-dir\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.786412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.784035    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-run-systemd\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.786412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.784037    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-sys\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.786412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.784053    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7-tmp-dir\") pod \"node-resolver-b2bnk\" (UID: \"b45c13d7-29c0-4054-9b7d-fd33ecec9cf7\") " pod="openshift-dns/node-resolver-b2bnk"
Oct 30 00:33:56.786412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.784052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.786412 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.784087    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-run-multus-certs\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.786917 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.784059    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-slash\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.786917 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.784087    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.786917 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.784100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/070879ac-09e8-4fdb-b55f-e52a9998552f-ovn-node-metrics-cert\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.786917 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:56.784135    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:33:56.786917 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.784177    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-socket-dir-parent\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.786917 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.784135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-socket-dir-parent\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.786917 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:56.784200    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret podName:b7f00b4a-044e-49ab-8f43-4351590692e8 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:57.284182207 +0000 UTC m=+2.156938430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret") pod "global-pull-secret-syncer-7vsmg" (UID: "b7f00b4a-044e-49ab-8f43-4351590692e8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:33:56.786917 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.784242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-kubernetes\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.786917 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.784298    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-kubernetes\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.786917 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.785268    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-tmp\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.786917 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.786018    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-tuned\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.786917 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.786115    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47d34735-e4b1-4073-b6de-98a9a720432b-agent-certs\") pod \"konnectivity-agent-v6rdh\" (UID: \"47d34735-e4b1-4073-b6de-98a9a720432b\") " pod="kube-system/konnectivity-agent-v6rdh"
Oct 30 00:33:56.786917 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.786213    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/070879ac-09e8-4fdb-b55f-e52a9998552f-ovn-node-metrics-cert\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.808766 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:56.808742    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:33:56.808766 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:56.808767    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:33:56.808935 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:56.808796    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qbhmk for pod openshift-network-diagnostics/network-check-target-62gpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:33:56.808935 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:56.808867    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk podName:c9803898-12ec-4a07-9a8f-1eef3f89235b nodeName:}" failed. No retries permitted until 2026-10-30 00:33:57.308852024 +0000 UTC m=+2.181608244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qbhmk" (UniqueName: "kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk") pod "network-check-target-62gpq" (UID: "c9803898-12ec-4a07-9a8f-1eef3f89235b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:33:56.811071 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.811043    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmqr7\" (UniqueName: \"kubernetes.io/projected/5ac72880-98ab-4093-b7aa-903a37c8113e-kube-api-access-bmqr7\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.811873 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.811847    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-257nk\" (UniqueName: \"kubernetes.io/projected/070879ac-09e8-4fdb-b55f-e52a9998552f-kube-api-access-257nk\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:56.811970 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.811848    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4tjg\" (UniqueName: \"kubernetes.io/projected/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7-kube-api-access-f4tjg\") pod \"node-resolver-b2bnk\" (UID: \"b45c13d7-29c0-4054-9b7d-fd33ecec9cf7\") " pod="openshift-dns/node-resolver-b2bnk"
Oct 30 00:33:56.811970 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.811956    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9rmw\" (UniqueName: \"kubernetes.io/projected/32086945-d23d-49bb-a876-fab4cef52ec6-kube-api-access-d9rmw\") pod \"iptables-alerter-whx68\" (UID: \"32086945-d23d-49bb-a876-fab4cef52ec6\") " pod="openshift-network-operator/iptables-alerter-whx68"
Oct 30 00:33:56.812056 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.811954    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nbn8\" (UniqueName: \"kubernetes.io/projected/3a145ece-ad48-43c2-a259-b9590802cfe8-kube-api-access-8nbn8\") pod \"network-metrics-daemon-wb6st\" (UID: \"3a145ece-ad48-43c2-a259-b9590802cfe8\") " pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:33:56.812056 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.812044    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f24vm\" (UniqueName: \"kubernetes.io/projected/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-kube-api-access-f24vm\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.812506 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.812491    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbtjm\" (UniqueName: \"kubernetes.io/projected/d996f136-ba95-4636-a870-94520d3eff0f-kube-api-access-vbtjm\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.815268 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.815254    2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:33:56.904937 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.904911    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:33:56.915508 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:56.915486    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733527bc_83b7_423c_a79e_1de279f5a607.slice/crio-6f3c807b145f40d120e4ae924c62641da716306c354344fb78151aabbea7d01e WatchSource:0}: Error finding container 6f3c807b145f40d120e4ae924c62641da716306c354344fb78151aabbea7d01e: Status 404 returned error can't find the container with id 6f3c807b145f40d120e4ae924c62641da716306c354344fb78151aabbea7d01e
Oct 30 00:33:56.916514 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.916495    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d9lhd"
Oct 30 00:33:56.926265 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:56.926245    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5975a832_beb3_4e76_98ad_aef36a2ccce2.slice/crio-a58bdd63ee44dcdfe20b21c5471578b608e708d7aaa3c13f4c20b5f89c390711 WatchSource:0}: Error finding container a58bdd63ee44dcdfe20b21c5471578b608e708d7aaa3c13f4c20b5f89c390711: Status 404 returned error can't find the container with id a58bdd63ee44dcdfe20b21c5471578b608e708d7aaa3c13f4c20b5f89c390711
Oct 30 00:33:56.948260 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.948239    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:33:56.956217 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:56.956196    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73f4c46_cb05_492e_8dbc_953f3c70b1d5.slice/crio-c67e4197c1f532fd42153c1cf0c45dfe1374d4c99d385bae87f9a63865323ea3 WatchSource:0}: Error finding container c67e4197c1f532fd42153c1cf0c45dfe1374d4c99d385bae87f9a63865323ea3: Status 404 returned error can't find the container with id c67e4197c1f532fd42153c1cf0c45dfe1374d4c99d385bae87f9a63865323ea3
Oct 30 00:33:56.969548 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.969500    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v6rdh"
Oct 30 00:33:56.972307 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.972292    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-42gzj"
Oct 30 00:33:56.977039 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.976858    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:33:56.980953 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:56.980933    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d34735_e4b1_4073_b6de_98a9a720432b.slice/crio-159cd984f3a29b117fdc8f93c0d301014dfef8c4b4ff50becdc1c6d7b6585fad WatchSource:0}: Error finding container 159cd984f3a29b117fdc8f93c0d301014dfef8c4b4ff50becdc1c6d7b6585fad: Status 404 returned error can't find the container with id 159cd984f3a29b117fdc8f93c0d301014dfef8c4b4ff50becdc1c6d7b6585fad
Oct 30 00:33:56.985361 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:56.985338    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac72880_98ab_4093_b7aa_903a37c8113e.slice/crio-8691cb9acb1869b233346dbd05a670fea0cf9c382094432b55288b91d83852c2 WatchSource:0}: Error finding container 8691cb9acb1869b233346dbd05a670fea0cf9c382094432b55288b91d83852c2: Status 404 returned error can't find the container with id 8691cb9acb1869b233346dbd05a670fea0cf9c382094432b55288b91d83852c2
Oct 30 00:33:56.986214 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:56.986193    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd996f136_ba95_4636_a870_94520d3eff0f.slice/crio-ccbb805d5abfb4915455ccbb72ea088e0952c14325b69d672c20b45e6fcec5c8 WatchSource:0}: Error finding container ccbb805d5abfb4915455ccbb72ea088e0952c14325b69d672c20b45e6fcec5c8: Status 404 returned error can't find the container with id ccbb805d5abfb4915455ccbb72ea088e0952c14325b69d672c20b45e6fcec5c8
Oct 30 00:33:56.989800 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.989769    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b2bnk"
Oct 30 00:33:56.991929 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.991910    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-whx68"
Oct 30 00:33:56.998225 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:56.998200    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:33:57.000222 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:57.000191    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45c13d7_29c0_4054_9b7d_fd33ecec9cf7.slice/crio-f1af4bcd067d00480932bbef7d3d95d8577e178a33a277d7ab5d7b1b8c1c0ddd WatchSource:0}: Error finding container f1af4bcd067d00480932bbef7d3d95d8577e178a33a277d7ab5d7b1b8c1c0ddd: Status 404 returned error can't find the container with id f1af4bcd067d00480932bbef7d3d95d8577e178a33a277d7ab5d7b1b8c1c0ddd
Oct 30 00:33:57.005579 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:57.005551    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32086945_d23d_49bb_a876_fab4cef52ec6.slice/crio-82479f721db5be7196aa5f19098e38b102f9fb744d55881c21733f3bdd59f635 WatchSource:0}: Error finding container 82479f721db5be7196aa5f19098e38b102f9fb744d55881c21733f3bdd59f635: Status 404 returned error can't find the container with id 82479f721db5be7196aa5f19098e38b102f9fb744d55881c21733f3bdd59f635
Oct 30 00:33:57.008476 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:33:57.008454    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070879ac_09e8_4fdb_b55f_e52a9998552f.slice/crio-adcc8ffdae79afac97367613656572dba910587bee16e642ad90bc6ff4f6fa93 WatchSource:0}: Error finding container adcc8ffdae79afac97367613656572dba910587bee16e642ad90bc6ff4f6fa93: Status 404 returned error can't find the container with id adcc8ffdae79afac97367613656572dba910587bee16e642ad90bc6ff4f6fa93
Oct 30 00:33:57.287890 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:57.287798    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:33:57.287890 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:57.287879    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs\") pod \"network-metrics-daemon-wb6st\" (UID: \"3a145ece-ad48-43c2-a259-b9590802cfe8\") " pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:33:57.288097 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:57.287925    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:33:57.288097 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:57.288006    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret podName:b7f00b4a-044e-49ab-8f43-4351590692e8 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:58.287987456 +0000 UTC m=+3.160743678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret") pod "global-pull-secret-syncer-7vsmg" (UID: "b7f00b4a-044e-49ab-8f43-4351590692e8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:33:57.288097 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:57.288018    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:33:57.288097 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:57.288070    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs podName:3a145ece-ad48-43c2-a259-b9590802cfe8 nodeName:}" failed. No retries permitted until 2026-10-30 00:33:58.288053235 +0000 UTC m=+3.160809444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs") pod "network-metrics-daemon-wb6st" (UID: "3a145ece-ad48-43c2-a259-b9590802cfe8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:33:57.389076 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:57.389030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhmk\" (UniqueName: \"kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk\") pod \"network-check-target-62gpq\" (UID: \"c9803898-12ec-4a07-9a8f-1eef3f89235b\") " pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:33:57.389812 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:57.389251    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:33:57.389812 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:57.389270    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:33:57.389812 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:57.389282    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qbhmk for pod openshift-network-diagnostics/network-check-target-62gpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:33:57.389812 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:57.389343    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk podName:c9803898-12ec-4a07-9a8f-1eef3f89235b nodeName:}" failed. No retries permitted until 2026-10-30 00:33:58.38932429 +0000 UTC m=+3.262080498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbhmk" (UniqueName: "kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk") pod "network-check-target-62gpq" (UID: "c9803898-12ec-4a07-9a8f-1eef3f89235b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:33:57.648155 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:57.648007    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-42gzj" event={"ID":"5ac72880-98ab-4093-b7aa-903a37c8113e","Type":"ContainerStarted","Data":"8691cb9acb1869b233346dbd05a670fea0cf9c382094432b55288b91d83852c2"}
Oct 30 00:33:57.655808 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:57.655739    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v6rdh" event={"ID":"47d34735-e4b1-4073-b6de-98a9a720432b","Type":"ContainerStarted","Data":"159cd984f3a29b117fdc8f93c0d301014dfef8c4b4ff50becdc1c6d7b6585fad"}
Oct 30 00:33:57.660067 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:57.659977    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" event={"ID":"553cfdac89599e545ab5a9539f3d3b39","Type":"ContainerStarted","Data":"edaebab344eb8b504f4cbb62b8b245ce7e6a502dfbe2011df04d7030ddd7e6e8"}
Oct 30 00:33:57.665518 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:57.665488    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc" event={"ID":"070879ac-09e8-4fdb-b55f-e52a9998552f","Type":"ContainerStarted","Data":"adcc8ffdae79afac97367613656572dba910587bee16e642ad90bc6ff4f6fa93"}
Oct 30 00:33:57.680499 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:57.680469    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-whx68" event={"ID":"32086945-d23d-49bb-a876-fab4cef52ec6","Type":"ContainerStarted","Data":"82479f721db5be7196aa5f19098e38b102f9fb744d55881c21733f3bdd59f635"}
Oct 30 00:33:57.695640 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:57.695575    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:28:57 +0000 UTC" deadline="2028-05-12 20:36:48.149020432 +0000 UTC"
Oct 30 00:33:57.695640 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:57.695640    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13460h2m50.453384627s"
Oct 30 00:33:57.703998 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:57.703968    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b2bnk" event={"ID":"b45c13d7-29c0-4054-9b7d-fd33ecec9cf7","Type":"ContainerStarted","Data":"f1af4bcd067d00480932bbef7d3d95d8577e178a33a277d7ab5d7b1b8c1c0ddd"}
Oct 30 00:33:57.722272 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:57.722225    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9h5hq" event={"ID":"b73f4c46-cb05-492e-8dbc-953f3c70b1d5","Type":"ContainerStarted","Data":"c67e4197c1f532fd42153c1cf0c45dfe1374d4c99d385bae87f9a63865323ea3"}
Oct 30 00:33:57.738744 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:57.738696    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d9lhd" event={"ID":"5975a832-beb3-4e76-98ad-aef36a2ccce2","Type":"ContainerStarted","Data":"a58bdd63ee44dcdfe20b21c5471578b608e708d7aaa3c13f4c20b5f89c390711"}
Oct 30 00:33:57.749769 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:57.749737    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v" event={"ID":"733527bc-83b7-423c-a79e-1de279f5a607","Type":"ContainerStarted","Data":"6f3c807b145f40d120e4ae924c62641da716306c354344fb78151aabbea7d01e"}
Oct 30 00:33:57.765222 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:57.765194    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal" event={"ID":"f8c3a9edbd8cbf03175beab8c4db7242","Type":"ContainerStarted","Data":"ab0c9e3bbd550a22c57907f05402ac9314905c0c759dec1a19a3abf77b32903e"}
Oct 30 00:33:57.779066 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:57.779005    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gmgzr" event={"ID":"d996f136-ba95-4636-a870-94520d3eff0f","Type":"ContainerStarted","Data":"ccbb805d5abfb4915455ccbb72ea088e0952c14325b69d672c20b45e6fcec5c8"}
Oct 30 00:33:57.861874 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:57.861842    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:33:58.297760 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:58.297709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:33:58.298030 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:58.297795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs\") pod \"network-metrics-daemon-wb6st\" (UID: \"3a145ece-ad48-43c2-a259-b9590802cfe8\") " pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:33:58.298030 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:58.297938    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:33:58.298030 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:58.298003    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs podName:3a145ece-ad48-43c2-a259-b9590802cfe8 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:00.297984261 +0000 UTC m=+5.170740478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs") pod "network-metrics-daemon-wb6st" (UID: "3a145ece-ad48-43c2-a259-b9590802cfe8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:33:58.298450 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:58.298426    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:33:58.298560 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:58.298486    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret podName:b7f00b4a-044e-49ab-8f43-4351590692e8 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:00.298471098 +0000 UTC m=+5.171227318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret") pod "global-pull-secret-syncer-7vsmg" (UID: "b7f00b4a-044e-49ab-8f43-4351590692e8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:33:58.400136 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:58.400100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhmk\" (UniqueName: \"kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk\") pod \"network-check-target-62gpq\" (UID: \"c9803898-12ec-4a07-9a8f-1eef3f89235b\") " pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:33:58.400726 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:58.400276    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:33:58.400726 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:58.400297    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:33:58.400726 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:58.400310    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qbhmk for pod openshift-network-diagnostics/network-check-target-62gpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:33:58.400726 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:58.400371    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk podName:c9803898-12ec-4a07-9a8f-1eef3f89235b nodeName:}" failed. No retries permitted until 2026-10-30 00:34:00.400352479 +0000 UTC m=+5.273108698 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbhmk" (UniqueName: "kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk") pod "network-check-target-62gpq" (UID: "c9803898-12ec-4a07-9a8f-1eef3f89235b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:33:58.605243 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:58.605141    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:33:58.605431 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:58.605287    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:33:58.605743 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:58.605718    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:33:58.605873 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:58.605851    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:33:58.605939 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:58.605929    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:33:58.606027 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:33:58.606008    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:33:58.696877 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:58.696787    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:28:57 +0000 UTC" deadline="2028-08-07 08:10:03.98658889 +0000 UTC"
Oct 30 00:33:58.696877 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:58.696827    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15535h36m5.289765344s"
Oct 30 00:33:59.083635 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:33:59.083574    2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:34:00.321259 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:00.320534    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:00.321259 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:00.320585    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs\") pod \"network-metrics-daemon-wb6st\" (UID: \"3a145ece-ad48-43c2-a259-b9590802cfe8\") " pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:00.321259 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:00.320700    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:34:00.321259 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:00.320747    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs podName:3a145ece-ad48-43c2-a259-b9590802cfe8 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:04.320735486 +0000 UTC m=+9.193491690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs") pod "network-metrics-daemon-wb6st" (UID: "3a145ece-ad48-43c2-a259-b9590802cfe8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:34:00.321259 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:00.321105    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:34:00.321259 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:00.321164    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret podName:b7f00b4a-044e-49ab-8f43-4351590692e8 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:04.321146007 +0000 UTC m=+9.193902229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret") pod "global-pull-secret-syncer-7vsmg" (UID: "b7f00b4a-044e-49ab-8f43-4351590692e8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:34:00.422634 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:00.421931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhmk\" (UniqueName: \"kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk\") pod \"network-check-target-62gpq\" (UID: \"c9803898-12ec-4a07-9a8f-1eef3f89235b\") " pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:00.422634 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:00.422123    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:34:00.422634 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:00.422143    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:34:00.422634 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:00.422157    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qbhmk for pod openshift-network-diagnostics/network-check-target-62gpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:34:00.422634 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:00.422246    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk podName:c9803898-12ec-4a07-9a8f-1eef3f89235b nodeName:}" failed. No retries permitted until 2026-10-30 00:34:04.422227049 +0000 UTC m=+9.294983255 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbhmk" (UniqueName: "kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk") pod "network-check-target-62gpq" (UID: "c9803898-12ec-4a07-9a8f-1eef3f89235b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:34:00.605444 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:00.605266    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:00.605444 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:00.605367    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:00.605755 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:00.605696    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:00.605854 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:00.605792    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:00.605854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:00.605842    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:00.605961 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:00.605917    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:01.818062 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:01.817315    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal" event={"ID":"f8c3a9edbd8cbf03175beab8c4db7242","Type":"ContainerStarted","Data":"1e5b061c695be95293eb2b31260a52d9a68b4a5c469a4ef5870b2e8bd8074406"}
Oct 30 00:34:01.826095 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:01.826066    2581 generic.go:356] "Generic (PLEG): container finished" podID="d996f136-ba95-4636-a870-94520d3eff0f" containerID="56a19c900ea14a288c773079a23d990e8ea27c13237434de39878047dea33c73" exitCode=0
Oct 30 00:34:01.826240 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:01.826151    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gmgzr" event={"ID":"d996f136-ba95-4636-a870-94520d3eff0f","Type":"ContainerDied","Data":"56a19c900ea14a288c773079a23d990e8ea27c13237434de39878047dea33c73"}
Oct 30 00:34:01.832324 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:01.832297    2581 generic.go:356] "Generic (PLEG): container finished" podID="553cfdac89599e545ab5a9539f3d3b39" containerID="00c9adad48ce7102684c8f7be5cc6c6245f7ff1de066b0ff9cc77dcf355bc510" exitCode=0
Oct 30 00:34:01.832411 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:01.832340    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" event={"ID":"553cfdac89599e545ab5a9539f3d3b39","Type":"ContainerDied","Data":"00c9adad48ce7102684c8f7be5cc6c6245f7ff1de066b0ff9cc77dcf355bc510"}
Oct 30 00:34:01.862660 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:01.862224    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal" podStartSLOduration=4.862205503 podStartE2EDuration="4.862205503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:01.86125191 +0000 UTC m=+6.734008153" watchObservedRunningTime="2026-10-30 00:34:01.862205503 +0000 UTC m=+6.734961731"
Oct 30 00:34:02.605546 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:02.605467    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:02.605698 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:02.605604    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:02.605698 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:02.605687    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:02.605844 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:02.605782    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:02.605844 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:02.605842    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:02.605950 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:02.605936    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:02.837130 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:02.837078    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v6rdh" event={"ID":"47d34735-e4b1-4073-b6de-98a9a720432b","Type":"ContainerStarted","Data":"8b9ff87c009431b71cb4f3608cf291d43e7102e3ff699a60f90e6cea7d8f8c6c"}
Oct 30 00:34:02.841550 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:02.841517    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" event={"ID":"553cfdac89599e545ab5a9539f3d3b39","Type":"ContainerStarted","Data":"7fc5fe31094c3b41c9d6235f865d53d0107e10e86778040607d24abbfa8a9908"}
Oct 30 00:34:02.845113 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:02.845085    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d9lhd" event={"ID":"5975a832-beb3-4e76-98ad-aef36a2ccce2","Type":"ContainerStarted","Data":"4676a93b7f64d91329eda534695b36c3a93f876454b52baf621eec9c73d8df95"}
Oct 30 00:34:02.936182 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:02.936082    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v6rdh" podStartSLOduration=2.249079976 podStartE2EDuration="6.936067312s" totalImagesPullingTime="4.686987336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:02.889752822 +0000 UTC m=+7.762509050" watchObservedRunningTime="2026-10-30 00:34:02.936067312 +0000 UTC m=+7.808823538"
Oct 30 00:34:02.936367 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:02.936287    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d9lhd" podStartSLOduration=1.519475879 podStartE2EDuration="6.936277045s" totalImagesPullingTime="5.416801166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:02.935186354 +0000 UTC m=+7.807942582" watchObservedRunningTime="2026-10-30 00:34:02.936277045 +0000 UTC m=+7.809033274"
Oct 30 00:34:02.984308 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:02.983829    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" podStartSLOduration=5.983809958 podStartE2EDuration="5.983809958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:02.982881264 +0000 UTC m=+7.855637492" watchObservedRunningTime="2026-10-30 00:34:02.983809958 +0000 UTC m=+7.856566186"
Oct 30 00:34:03.112824 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:03.112454    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v6rdh"
Oct 30 00:34:03.113627 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:03.113426    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v6rdh"
Oct 30 00:34:03.851980 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:03.851949    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v" event={"ID":"733527bc-83b7-423c-a79e-1de279f5a607","Type":"ContainerStarted","Data":"d2a4e81ea2b12c2e8af7e8467cb15e7c035ad429fb7ff54ec4ae7a40743374ea"}
Oct 30 00:34:04.351397 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:04.351358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs\") pod \"network-metrics-daemon-wb6st\" (UID: \"3a145ece-ad48-43c2-a259-b9590802cfe8\") " pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:04.351568 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:04.351442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:04.351649 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:04.351567    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:34:04.351649 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:04.351633    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret podName:b7f00b4a-044e-49ab-8f43-4351590692e8 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:12.351613509 +0000 UTC m=+17.224369716 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret") pod "global-pull-secret-syncer-7vsmg" (UID: "b7f00b4a-044e-49ab-8f43-4351590692e8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:34:04.352237 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:04.352068    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:34:04.352237 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:04.352123    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs podName:3a145ece-ad48-43c2-a259-b9590802cfe8 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:12.352106744 +0000 UTC m=+17.224862950 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs") pod "network-metrics-daemon-wb6st" (UID: "3a145ece-ad48-43c2-a259-b9590802cfe8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:34:04.452075 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:04.451998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhmk\" (UniqueName: \"kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk\") pod \"network-check-target-62gpq\" (UID: \"c9803898-12ec-4a07-9a8f-1eef3f89235b\") " pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:04.452226 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:04.452193    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:34:04.452226 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:04.452221    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:34:04.452346 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:04.452235    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qbhmk for pod openshift-network-diagnostics/network-check-target-62gpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:34:04.452346 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:04.452293    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk podName:c9803898-12ec-4a07-9a8f-1eef3f89235b nodeName:}" failed. No retries permitted until 2026-10-30 00:34:12.452274538 +0000 UTC m=+17.325030756 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbhmk" (UniqueName: "kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk") pod "network-check-target-62gpq" (UID: "c9803898-12ec-4a07-9a8f-1eef3f89235b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:34:04.605700 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:04.605662    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:04.605876 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:04.605831    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:04.606325 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:04.606301    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:04.606439 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:04.606419    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:04.606511 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:04.606497    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:04.606594 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:04.606577    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:04.854039 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:04.854010    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:34:05.272784 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:05.272749    2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 00:34:05.550339 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:05.550232    2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:34:05.27277706Z","UUID":"8d08a769-48fb-45eb-98ab-f25fee502281","Handler":null,"Name":"","Endpoint":""}
Oct 30 00:34:05.552693 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:05.552667    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 00:34:05.552693 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:05.552701    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 00:34:05.860076 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:05.859932    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v" event={"ID":"733527bc-83b7-423c-a79e-1de279f5a607","Type":"ContainerStarted","Data":"2fd5c272f397e419523aa5848264aab6722eb177d383077c8d82af89d8e40f04"}
Oct 30 00:34:05.864145 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:05.864107    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-42gzj" event={"ID":"5ac72880-98ab-4093-b7aa-903a37c8113e","Type":"ContainerStarted","Data":"821ee53d1686dced11badbe72c1340999710d6e13d6ff8773cac35ed0e0ec554"}
Oct 30 00:34:05.867551 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:05.867523    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b2bnk" event={"ID":"b45c13d7-29c0-4054-9b7d-fd33ecec9cf7","Type":"ContainerStarted","Data":"9838da3a9b4afdb7560d9feae973dda12b0fc311a49641a1800dd8d97c932092"}
Oct 30 00:34:05.959779 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:05.959670    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-b2bnk" podStartSLOduration=1.3935849980000001 podStartE2EDuration="9.959651497s" totalImagesPullingTime="8.566066499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:05.95934547 +0000 UTC m=+10.832101697" watchObservedRunningTime="2026-10-30 00:34:05.959651497 +0000 UTC m=+10.832407723"
Oct 30 00:34:05.959931 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:05.959851    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-42gzj" podStartSLOduration=1.567256894 podStartE2EDuration="9.959843994s" totalImagesPullingTime="8.3925871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:05.914429959 +0000 UTC m=+10.787186188" watchObservedRunningTime="2026-10-30 00:34:05.959843994 +0000 UTC m=+10.832600220"
Oct 30 00:34:06.605298 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:06.605271    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:06.605468 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:06.605361    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:06.605537 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:06.605499    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:06.605537 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:06.605509    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:06.605639 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:06.605615    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:06.605751 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:06.605715    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:06.873117 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:06.872911    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v" event={"ID":"733527bc-83b7-423c-a79e-1de279f5a607","Type":"ContainerStarted","Data":"2ddfb192c84fca1a804cf76cc5f697f9d982fc31e9517211188886ebc2968eff"}
Oct 30 00:34:06.875174 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:06.875148    2581 generic.go:356] "Generic (PLEG): container finished" podID="d996f136-ba95-4636-a870-94520d3eff0f" containerID="8ad112f314388e3afc623fc4b39d7bd8de911433551c542a1dd52b61f5dc9ce7" exitCode=0
Oct 30 00:34:06.875457 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:06.875434    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gmgzr" event={"ID":"d996f136-ba95-4636-a870-94520d3eff0f","Type":"ContainerDied","Data":"8ad112f314388e3afc623fc4b39d7bd8de911433551c542a1dd52b61f5dc9ce7"}
Oct 30 00:34:06.922444 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:06.922396    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v" podStartSLOduration=1.8681305959999999 podStartE2EDuration="10.922380724s" totalImagesPullingTime="9.054250128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:06.921662324 +0000 UTC m=+11.794418566" watchObservedRunningTime="2026-10-30 00:34:06.922380724 +0000 UTC m=+11.795136950"
Oct 30 00:34:07.879797 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:07.879757    2581 generic.go:356] "Generic (PLEG): container finished" podID="d996f136-ba95-4636-a870-94520d3eff0f" containerID="8205418c25c455ff29f3f6f0916f344b493f31b5f5207986d8742398e359fe58" exitCode=0
Oct 30 00:34:07.880290 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:07.879842    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gmgzr" event={"ID":"d996f136-ba95-4636-a870-94520d3eff0f","Type":"ContainerDied","Data":"8205418c25c455ff29f3f6f0916f344b493f31b5f5207986d8742398e359fe58"}
Oct 30 00:34:07.881538 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:07.881505    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-whx68" event={"ID":"32086945-d23d-49bb-a876-fab4cef52ec6","Type":"ContainerStarted","Data":"bb6390c4959b45bc3a7cd5daf7ed547465aee0f6b78d6683fe25d005342ccf29"}
Oct 30 00:34:07.983381 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:07.983334    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-whx68" podStartSLOduration=3.027488452 podStartE2EDuration="11.983319683s" totalImagesPullingTime="8.955831231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:07.982975438 +0000 UTC m=+12.855731662" watchObservedRunningTime="2026-10-30 00:34:07.983319683 +0000 UTC m=+12.856075905"
Oct 30 00:34:08.605390 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:08.605345    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:08.605390 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:08.605371    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:08.605602 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:08.605362    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:08.605602 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:08.605470    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:08.605602 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:08.605547    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:08.605777 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:08.605628    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:08.885813 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:08.885726    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9h5hq" event={"ID":"b73f4c46-cb05-492e-8dbc-953f3c70b1d5","Type":"ContainerStarted","Data":"17e925205bae2f17fbdc294574526e08a61fc3ad53c6288ccf71df65cbb56549"}
Oct 30 00:34:08.930638 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:08.930589    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9h5hq" podStartSLOduration=1.677228247 podStartE2EDuration="12.93057339s" totalImagesPullingTime="11.253345143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:08.930037945 +0000 UTC m=+13.802794168" watchObservedRunningTime="2026-10-30 00:34:08.93057339 +0000 UTC m=+13.803329614"
Oct 30 00:34:09.890381 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:09.890349    2581 generic.go:356] "Generic (PLEG): container finished" podID="d996f136-ba95-4636-a870-94520d3eff0f" containerID="e73e2077a4c8792ae759ea515baf233207dba26ab14d2f08a27dadfdf6f05003" exitCode=0
Oct 30 00:34:09.890904 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:09.890399    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gmgzr" event={"ID":"d996f136-ba95-4636-a870-94520d3eff0f","Type":"ContainerDied","Data":"e73e2077a4c8792ae759ea515baf233207dba26ab14d2f08a27dadfdf6f05003"}
Oct 30 00:34:10.605265 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:10.605229    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:10.605265 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:10.605260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:10.605481 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:10.605366    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:10.605481 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:10.605391    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:10.605568 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:10.605481    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:10.605615 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:10.605562    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:11.898627 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:11.898441    2581 generic.go:356] "Generic (PLEG): container finished" podID="d996f136-ba95-4636-a870-94520d3eff0f" containerID="8e83bb8f0891b297a06d182c99dec138c964d7fba95fefe6859c771bbd0dc4e9" exitCode=0
Oct 30 00:34:11.898627 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:11.898518    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gmgzr" event={"ID":"d996f136-ba95-4636-a870-94520d3eff0f","Type":"ContainerDied","Data":"8e83bb8f0891b297a06d182c99dec138c964d7fba95fefe6859c771bbd0dc4e9"}
Oct 30 00:34:12.412280 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:12.412247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:12.412384 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:12.412337    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs\") pod \"network-metrics-daemon-wb6st\" (UID: \"3a145ece-ad48-43c2-a259-b9590802cfe8\") " pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:12.412448 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:12.412386    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:34:12.412448 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:12.412431    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:34:12.412448 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:12.412439    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret podName:b7f00b4a-044e-49ab-8f43-4351590692e8 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:28.412426337 +0000 UTC m=+33.285182541 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret") pod "global-pull-secret-syncer-7vsmg" (UID: "b7f00b4a-044e-49ab-8f43-4351590692e8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:34:12.412603 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:12.412478    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs podName:3a145ece-ad48-43c2-a259-b9590802cfe8 nodeName:}" failed. No retries permitted until 2026-10-30 00:34:28.412466399 +0000 UTC m=+33.285222601 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs") pod "network-metrics-daemon-wb6st" (UID: "3a145ece-ad48-43c2-a259-b9590802cfe8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:34:12.513107 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:12.513081    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhmk\" (UniqueName: \"kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk\") pod \"network-check-target-62gpq\" (UID: \"c9803898-12ec-4a07-9a8f-1eef3f89235b\") " pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:12.513245 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:12.513229    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:34:12.513278 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:12.513249    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:34:12.513278 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:12.513259    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qbhmk for pod openshift-network-diagnostics/network-check-target-62gpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:34:12.513336 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:12.513306    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk podName:c9803898-12ec-4a07-9a8f-1eef3f89235b nodeName:}" failed. No retries permitted until 2026-10-30 00:34:28.513287563 +0000 UTC m=+33.386043776 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbhmk" (UniqueName: "kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk") pod "network-check-target-62gpq" (UID: "c9803898-12ec-4a07-9a8f-1eef3f89235b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:34:12.605617 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:12.605592    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:12.605722 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:12.605662    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:12.605722 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:12.605685    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:12.605803 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:12.605770    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:12.605844 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:12.605828    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:12.605904 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:12.605882    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:12.903890 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:12.903861    2581 generic.go:356] "Generic (PLEG): container finished" podID="d996f136-ba95-4636-a870-94520d3eff0f" containerID="72be51cb9ed7eff16bc20d3f12939abdace8727c78b193f4dda082cb5469776f" exitCode=0
Oct 30 00:34:12.904257 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:12.903940    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gmgzr" event={"ID":"d996f136-ba95-4636-a870-94520d3eff0f","Type":"ContainerDied","Data":"72be51cb9ed7eff16bc20d3f12939abdace8727c78b193f4dda082cb5469776f"}
Oct 30 00:34:12.906696 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:12.906676    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 00:34:12.907002 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:12.906985    2581 generic.go:356] "Generic (PLEG): container finished" podID="070879ac-09e8-4fdb-b55f-e52a9998552f" containerID="59428dffec1172c34500f519af0fb69b41510655bbd9b5635cb46d0a5324ac60" exitCode=1
Oct 30 00:34:12.907063 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:12.907012    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc" event={"ID":"070879ac-09e8-4fdb-b55f-e52a9998552f","Type":"ContainerStarted","Data":"534536b3d40a4342f6043d3f4ccf0f3faa1e114397f76bb41d1081ce7fea3142"}
Oct 30 00:34:12.907099 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:12.907065    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc" event={"ID":"070879ac-09e8-4fdb-b55f-e52a9998552f","Type":"ContainerStarted","Data":"0b976522dbfb785d4f83c3752fb62e94e89edcf019dc48e01e49a93e73c0b067"}
Oct 30 00:34:12.907099 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:12.907094    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc" event={"ID":"070879ac-09e8-4fdb-b55f-e52a9998552f","Type":"ContainerStarted","Data":"9a242fa3428fbc56c5746d208e3cbe8937e058197b38f2684c67da08c7c61c47"}
Oct 30 00:34:12.907151 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:12.907120    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc" event={"ID":"070879ac-09e8-4fdb-b55f-e52a9998552f","Type":"ContainerStarted","Data":"abae279fe62d083225b93d6a9413dfa840031c6a4b19f2a6c9ace46753271631"}
Oct 30 00:34:12.907151 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:12.907147    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc" event={"ID":"070879ac-09e8-4fdb-b55f-e52a9998552f","Type":"ContainerDied","Data":"59428dffec1172c34500f519af0fb69b41510655bbd9b5635cb46d0a5324ac60"}
Oct 30 00:34:12.907209 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:12.907175    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc" event={"ID":"070879ac-09e8-4fdb-b55f-e52a9998552f","Type":"ContainerStarted","Data":"280eff65285d25c6a115d9be9142a0922201401aac564ea9ce2d3eded51de7f7"}
Oct 30 00:34:13.912363 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:13.912326    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gmgzr" event={"ID":"d996f136-ba95-4636-a870-94520d3eff0f","Type":"ContainerStarted","Data":"cf435f39de166b6df7626ba8748186598c5814ce1257faedea54d39537a8cc03"}
Oct 30 00:34:13.989061 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:13.989010    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gmgzr" podStartSLOduration=6.937348703 podStartE2EDuration="17.988998433s" totalImagesPullingTime="11.05164973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:13.975258423 +0000 UTC m=+18.848014645" watchObservedRunningTime="2026-10-30 00:34:13.988998433 +0000 UTC m=+18.861754655"
Oct 30 00:34:14.605491 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:14.605454    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:14.605729 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:14.605555    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:14.605729 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:14.605578    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:14.605729 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:14.605571    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:14.605729 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:14.605644    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:14.605900 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:14.605786    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:14.917139 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:14.917066    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 00:34:14.917502 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:14.917480    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc" event={"ID":"070879ac-09e8-4fdb-b55f-e52a9998552f","Type":"ContainerStarted","Data":"0c1dff477b81936ba36e03bfa78364ed1f32ebda253e9a7ec3ec57d22eb2b8e2"}
Oct 30 00:34:16.605892 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:16.605852    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:16.605892 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:16.605886    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:16.606424 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:16.606000    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:16.606424 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:16.606018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:16.606424 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:16.606103    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:16.606424 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:16.606203    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:16.925691 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:16.925626    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 00:34:16.926048 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:16.926024    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc" event={"ID":"070879ac-09e8-4fdb-b55f-e52a9998552f","Type":"ContainerStarted","Data":"fb578b202d412e539d75f333f0ce287fe1564dfad8d0874b8996b8eaae2a9bb7"}
Oct 30 00:34:16.926265 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:16.926249    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:34:16.926391 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:16.926369    2581 scope.go:122] "RemoveContainer" containerID="59428dffec1172c34500f519af0fb69b41510655bbd9b5635cb46d0a5324ac60"
Oct 30 00:34:16.945409 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:16.945392    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:34:17.640100 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:17.639933    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:34:17.931594 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:17.931525    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 00:34:17.931911 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:17.931883    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc" event={"ID":"070879ac-09e8-4fdb-b55f-e52a9998552f","Type":"ContainerStarted","Data":"92738dccbb244326f9dc19eef561bf3d09c39c976f647468afecc833eb394bba"}
Oct 30 00:34:17.932075 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:17.932058    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:34:17.948605 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:17.948574    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:34:18.036273 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:18.036209    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc" podStartSLOduration=6.824476945 podStartE2EDuration="22.036191137s" totalImagesPullingTime="15.211714192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:18.030800544 +0000 UTC m=+22.903556769" watchObservedRunningTime="2026-10-30 00:34:18.036191137 +0000 UTC m=+22.908947364"
Oct 30 00:34:18.605143 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:18.605113    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:18.605268 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:18.605187    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:18.605303 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:18.605271    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:18.605303 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:18.605284    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:18.605384 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:18.605361    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:18.605423 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:18.605388    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:18.875350 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:18.875240    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wb6st"]
Oct 30 00:34:18.877359 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:18.877333    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-62gpq"]
Oct 30 00:34:18.877804 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:18.877784    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7vsmg"]
Oct 30 00:34:18.933830 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:18.933806    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:18.933937 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:18.933806    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:18.933983 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:18.933929    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:18.933983 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:18.933806    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:18.934046 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:18.933984    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:18.934046 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:18.934026    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:20.605140 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:20.605109    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:20.605562 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:20.605201    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:20.605562 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:20.605215    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:20.605562 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:20.605288    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:20.605562 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:20.605295    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:20.605562 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:20.605388    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:20.729106 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:20.729078    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v6rdh"
Oct 30 00:34:20.729260 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:20.729180    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:34:20.729598 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:20.729580    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v6rdh"
Oct 30 00:34:22.605294 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:22.605257    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:22.605294 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:22.605284    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:22.605843 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:22.605436    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:22.605843 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:22.605471    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:22.605843 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:22.605535    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:22.605843 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:22.605590    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:22.808934 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:22.808881    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v6rdh_47d34735-e4b1-4073-b6de-98a9a720432b/konnectivity-agent/0.log" limit=1024
Oct 30 00:34:24.115465 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:24.115434    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b2bnk_b45c13d7-29c0-4054-9b7d-fd33ecec9cf7/dns-node-resolver/0.log"
Oct 30 00:34:24.605336 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:24.605302    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:24.605509 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:24.605340    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:24.605509 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:24.605397    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:24.605509 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:24.605409    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:24.605509 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:24.605488    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:24.605661 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:24.605544    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:25.115146 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:25.115114    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d9lhd_5975a832-beb3-4e76-98ad-aef36a2ccce2/node-ca/0.log"
Oct 30 00:34:26.605435 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:26.605402    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:26.605840 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:26.605462    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:26.605840 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:26.605485    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:26.605840 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:26.605594    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:26.605840 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:26.605620    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:26.605840 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:26.605686    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:28.435313 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:28.435280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs\") pod \"network-metrics-daemon-wb6st\" (UID: \"3a145ece-ad48-43c2-a259-b9590802cfe8\") " pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:28.435681 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:28.435326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:28.435681 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:28.435428    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:34:28.435681 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:28.435476    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret podName:b7f00b4a-044e-49ab-8f43-4351590692e8 nodeName:}" failed. No retries permitted until 2026-10-30 00:35:00.435462675 +0000 UTC m=+65.308218875 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret") pod "global-pull-secret-syncer-7vsmg" (UID: "b7f00b4a-044e-49ab-8f43-4351590692e8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:34:28.435681 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:28.435428    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:34:28.435681 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:28.435549    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs podName:3a145ece-ad48-43c2-a259-b9590802cfe8 nodeName:}" failed. No retries permitted until 2026-10-30 00:35:00.435537255 +0000 UTC m=+65.308293470 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs") pod "network-metrics-daemon-wb6st" (UID: "3a145ece-ad48-43c2-a259-b9590802cfe8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:34:28.535664 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:28.535633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhmk\" (UniqueName: \"kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk\") pod \"network-check-target-62gpq\" (UID: \"c9803898-12ec-4a07-9a8f-1eef3f89235b\") " pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:28.535811 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:28.535789    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:34:28.535811 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:28.535804    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:34:28.535881 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:28.535813    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qbhmk for pod openshift-network-diagnostics/network-check-target-62gpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:34:28.535881 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:28.535860    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk podName:c9803898-12ec-4a07-9a8f-1eef3f89235b nodeName:}" failed. No retries permitted until 2026-10-30 00:35:00.53584427 +0000 UTC m=+65.408600482 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbhmk" (UniqueName: "kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk") pod "network-check-target-62gpq" (UID: "c9803898-12ec-4a07-9a8f-1eef3f89235b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:34:28.605697 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:28.605676    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:28.605824 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:28.605677    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:28.605882 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:28.605862    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:28.605930 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:28.605780    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:28.605930 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:28.605677    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:28.605999 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:28.605941    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:30.605870 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:30.605834    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:30.605870 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:30.605867    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:30.606267 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:30.605880    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:30.606267 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:30.605951    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:30.606267 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:30.606063    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:30.606267 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:30.606135    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:32.605195 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:32.605166    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:32.605599 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:32.605166    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:32.605599 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:32.605264    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:32.605599 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:32.605167    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:32.605599 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:32.605359    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:32.605599 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:32.605375    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:34.605147 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:34.605111    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:34.605518 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:34.605172    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:34.605518 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:34.605267    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:34.605518 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:34.605311    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:34.605518 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:34.605410    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:34.605518 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:34.605501    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:36.605097 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:36.605069    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:36.605097 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:36.605079    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:36.605564 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:36.605079    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:36.605564 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:36.605158    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vsmg" podUID="b7f00b4a-044e-49ab-8f43-4351590692e8"
Oct 30 00:34:36.605564 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:36.605225    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb6st" podUID="3a145ece-ad48-43c2-a259-b9590802cfe8"
Oct 30 00:34:36.605564 ip-10-0-130-245 kubenswrapper[2581]: E1030 00:34:36.605312    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62gpq" podUID="c9803898-12ec-4a07-9a8f-1eef3f89235b"
Oct 30 00:34:36.910681 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:36.910494    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeReady"
Oct 30 00:34:36.910813 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:36.910749    2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 00:34:36.970097 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:36.970074    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rwwwf"]
Oct 30 00:34:36.989668 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:36.989649    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wvmfk"]
Oct 30 00:34:36.989797 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:36.989778    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:34:37.005590 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.005567    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rwwwf"]
Oct 30 00:34:37.005728 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.005607    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xfk62"]
Oct 30 00:34:37.005728 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.005668    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:34:37.005854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.005789    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 00:34:37.005854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.005790    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-75g57\""
Oct 30 00:34:37.005854 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.005790    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 00:34:37.017775 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.017757    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wvmfk"]
Oct 30 00:34:37.017853 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.017839    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xfk62"
Oct 30 00:34:37.021952 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.021933    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xfk62"]
Oct 30 00:34:37.097460 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.097435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzmdn\" (UniqueName: \"kubernetes.io/projected/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-kube-api-access-xzmdn\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:34:37.097576 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.097468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-config-volume\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:34:37.097576 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.097494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-crio-socket\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:34:37.097665 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.097624    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpbnj\" (UniqueName: \"kubernetes.io/projected/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-kube-api-access-fpbnj\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:34:37.097665 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.097649    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-tmp-dir\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:34:37.097785 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.097669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-metrics-tls\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:34:37.097785 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.097688    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-data-volume\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:34:37.097785 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.097746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:34:37.097921 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.097850    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:34:37.106006 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.105986    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 00:34:37.106124 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.106052    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 00:34:37.106124 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.106080    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 00:34:37.106376 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.106136    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 00:34:37.106376 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.106188    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 00:34:37.106376 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.106238    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 00:34:37.106376 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.106270    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c889x\""
Oct 30 00:34:37.106376 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.106289    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pkqjh\""
Oct 30 00:34:37.106376 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.106305    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 00:34:37.198109 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.198050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-metrics-tls\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:34:37.198109 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.198088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-data-volume\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:34:37.198270 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.198112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:34:37.198270 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.198153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:34:37.198270 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.198207    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzmdn\" (UniqueName: \"kubernetes.io/projected/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-kube-api-access-xzmdn\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:34:37.198270 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.198232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-config-volume\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:34:37.198461 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.198391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b308ab3-9eb7-4be8-964f-ff94af40b883-cert\") pod \"ingress-canary-xfk62\" (UID: \"0b308ab3-9eb7-4be8-964f-ff94af40b883\") " pod="openshift-ingress-canary/ingress-canary-xfk62"
Oct 30 00:34:37.198461 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.198435    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-data-volume\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:34:37.198566 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.198515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-crio-socket\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:34:37.198620 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.198563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpbnj\" (UniqueName: \"kubernetes.io/projected/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-kube-api-access-fpbnj\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:34:37.198671 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.198612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j86q\" (UniqueName: \"kubernetes.io/projected/0b308ab3-9eb7-4be8-964f-ff94af40b883-kube-api-access-6j86q\") pod \"ingress-canary-xfk62\" (UID: \"0b308ab3-9eb7-4be8-964f-ff94af40b883\") " pod="openshift-ingress-canary/ingress-canary-xfk62"
Oct 30 00:34:37.198768 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.198646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-tmp-dir\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:34:37.198768 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.198724    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-crio-socket\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:34:37.198873 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.198830    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-config-volume\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:34:37.198873 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.198834    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:34:37.198962 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.198946    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-tmp-dir\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:34:37.203090 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.203068    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:34:37.203090 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.203086    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-metrics-tls\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:34:37.222754 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.222727    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzmdn\" (UniqueName: \"kubernetes.io/projected/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-kube-api-access-xzmdn\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:34:37.222905 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.222891    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpbnj\" (UniqueName: \"kubernetes.io/projected/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-kube-api-access-fpbnj\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:34:37.299945 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.299920    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b308ab3-9eb7-4be8-964f-ff94af40b883-cert\") pod \"ingress-canary-xfk62\" (UID: \"0b308ab3-9eb7-4be8-964f-ff94af40b883\") " pod="openshift-ingress-canary/ingress-canary-xfk62"
Oct 30 00:34:37.300034 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.299958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j86q\" (UniqueName: \"kubernetes.io/projected/0b308ab3-9eb7-4be8-964f-ff94af40b883-kube-api-access-6j86q\") pod \"ingress-canary-xfk62\" (UID: \"0b308ab3-9eb7-4be8-964f-ff94af40b883\") " pod="openshift-ingress-canary/ingress-canary-xfk62"
Oct 30 00:34:37.301992 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.301975    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b308ab3-9eb7-4be8-964f-ff94af40b883-cert\") pod \"ingress-canary-xfk62\" (UID: \"0b308ab3-9eb7-4be8-964f-ff94af40b883\") " pod="openshift-ingress-canary/ingress-canary-xfk62"
Oct 30 00:34:37.324143 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.324121    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j86q\" (UniqueName: \"kubernetes.io/projected/0b308ab3-9eb7-4be8-964f-ff94af40b883-kube-api-access-6j86q\") pod \"ingress-canary-xfk62\" (UID: \"0b308ab3-9eb7-4be8-964f-ff94af40b883\") " pod="openshift-ingress-canary/ingress-canary-xfk62"
Oct 30 00:34:37.324225 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.324205    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:34:37.368818 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.368802    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:34:37.382272 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.382253    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xfk62"
Oct 30 00:34:37.705170 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.705145    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xfk62"]
Oct 30 00:34:37.707377 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:34:37.707349    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b308ab3_9eb7_4be8_964f_ff94af40b883.slice/crio-e0abdd59e265409683f86e9f7c116e9998805263b62af79330f4dd475ca46a13 WatchSource:0}: Error finding container e0abdd59e265409683f86e9f7c116e9998805263b62af79330f4dd475ca46a13: Status 404 returned error can't find the container with id e0abdd59e265409683f86e9f7c116e9998805263b62af79330f4dd475ca46a13
Oct 30 00:34:37.708208 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.708188    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rwwwf"]
Oct 30 00:34:37.709134 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.709111    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wvmfk"]
Oct 30 00:34:37.709227 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:34:37.709152    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c3a600_551a_4ff4_8fbe_e9a2aa1172e3.slice/crio-2cd2a81a83b1742fb463233f235feabce5febdf65814c37354b4c9f36734834a WatchSource:0}: Error finding container 2cd2a81a83b1742fb463233f235feabce5febdf65814c37354b4c9f36734834a: Status 404 returned error can't find the container with id 2cd2a81a83b1742fb463233f235feabce5febdf65814c37354b4c9f36734834a
Oct 30 00:34:37.710054 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:34:37.710032    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2991faa6_6a5e_4fbb_8b0b_f17daa3d38a6.slice/crio-544c576d92b15de7c0ede662e9e7824f6378d9bbe565085dad652bcb21bfdce7 WatchSource:0}: Error finding container 544c576d92b15de7c0ede662e9e7824f6378d9bbe565085dad652bcb21bfdce7: Status 404 returned error can't find the container with id 544c576d92b15de7c0ede662e9e7824f6378d9bbe565085dad652bcb21bfdce7
Oct 30 00:34:37.973459 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.973224    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvmfk" event={"ID":"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6","Type":"ContainerStarted","Data":"63e4150e83158e0c0a617334889c2702e8e8c6d9e7cb637ac1cf00e5df1de84f"}
Oct 30 00:34:37.973459 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.973435    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvmfk" event={"ID":"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6","Type":"ContainerStarted","Data":"544c576d92b15de7c0ede662e9e7824f6378d9bbe565085dad652bcb21bfdce7"}
Oct 30 00:34:37.974279 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.974259    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xfk62" event={"ID":"0b308ab3-9eb7-4be8-964f-ff94af40b883","Type":"ContainerStarted","Data":"e0abdd59e265409683f86e9f7c116e9998805263b62af79330f4dd475ca46a13"}
Oct 30 00:34:37.975238 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:37.975221    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rwwwf" event={"ID":"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3","Type":"ContainerStarted","Data":"2cd2a81a83b1742fb463233f235feabce5febdf65814c37354b4c9f36734834a"}
Oct 30 00:34:38.605850 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:38.605818    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:34:38.605850 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:38.605842    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:34:38.606087 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:38.605845    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:34:38.622601 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:38.622580    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 00:34:38.622725 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:38.622682    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 00:34:38.636772 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:38.636751    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-78zxw\""
Oct 30 00:34:38.636874 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:38.636806    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q972j\""
Oct 30 00:34:38.636874 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:38.636833    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 00:34:38.636989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:38.636891    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 00:34:38.636989 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:38.636920    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-smldl\""
Oct 30 00:34:38.980470 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:38.980433    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvmfk" event={"ID":"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6","Type":"ContainerStarted","Data":"4baa98a9ee17020d61cda87513f22a8b948a78fdeb5db0091589202b16e1dc1d"}
Oct 30 00:34:39.985456 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:39.985381    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rwwwf" event={"ID":"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3","Type":"ContainerStarted","Data":"1725a4c85cadd4553c19654a183a64816d0bfe209cce92408b1a9e29719af496"}
Oct 30 00:34:39.985456 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:39.985446    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rwwwf" event={"ID":"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3","Type":"ContainerStarted","Data":"dbd991686974b14c6d9c21475ea95955c2f3b4e5fa2ce8760e6e1a2fea833bf6"}
Oct 30 00:34:39.986257 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:39.986237    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:34:40.031174 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.031125    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-rwwwf" podStartSLOduration=2.090247954 podStartE2EDuration="4.031111105s" totalImagesPullingTime="1.940863151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:40.030177054 +0000 UTC m=+44.902933297" watchObservedRunningTime="2026-10-30 00:34:40.031111105 +0000 UTC m=+44.903867328"
Oct 30 00:34:40.131426 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.131398    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"]
Oct 30 00:34:40.133299 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.133281    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:34:40.150959 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.150898    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 00:34:40.150959 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.150917    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 00:34:40.150959 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.150903    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 00:34:40.150959 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.150903    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 00:34:40.151192 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.151046    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mkc6n\""
Oct 30 00:34:40.151192 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.151130    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 00:34:40.155596 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.155576    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"]
Oct 30 00:34:40.319896 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.319864    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/21be6874-6138-4a00-9c6e-aa1841eb3230-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:34:40.319896 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.319905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7lkm\" (UniqueName: \"kubernetes.io/projected/21be6874-6138-4a00-9c6e-aa1841eb3230-kube-api-access-p7lkm\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:34:40.320147 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.320028    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21be6874-6138-4a00-9c6e-aa1841eb3230-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:34:40.320147 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.320064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21be6874-6138-4a00-9c6e-aa1841eb3230-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:34:40.420553 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.420512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/21be6874-6138-4a00-9c6e-aa1841eb3230-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:34:40.420553 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.420557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7lkm\" (UniqueName: \"kubernetes.io/projected/21be6874-6138-4a00-9c6e-aa1841eb3230-kube-api-access-p7lkm\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:34:40.420775 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.420613    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21be6874-6138-4a00-9c6e-aa1841eb3230-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:34:40.420775 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.420635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21be6874-6138-4a00-9c6e-aa1841eb3230-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:34:40.421695 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.421669    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21be6874-6138-4a00-9c6e-aa1841eb3230-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:34:40.424740 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.424579    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21be6874-6138-4a00-9c6e-aa1841eb3230-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:34:40.425192 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.425168    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/21be6874-6138-4a00-9c6e-aa1841eb3230-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:34:40.445128 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.445108    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7lkm\" (UniqueName: \"kubernetes.io/projected/21be6874-6138-4a00-9c6e-aa1841eb3230-kube-api-access-p7lkm\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:34:40.445231 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.445186    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:34:40.742077 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.742052    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"]
Oct 30 00:34:40.988876 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.988785    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj" event={"ID":"21be6874-6138-4a00-9c6e-aa1841eb3230","Type":"ContainerStarted","Data":"ad53fa1332f37c8b6761caff50c30eba5a7ae08ad8928e0fcbdc157d85629928"}
Oct 30 00:34:40.990314 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.990280    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xfk62" event={"ID":"0b308ab3-9eb7-4be8-964f-ff94af40b883","Type":"ContainerStarted","Data":"c75ce77fbd90af2f797bceb8d9af5209eeebb34aa1e1786470ee797a42ce0086"}
Oct 30 00:34:40.992249 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:40.992222    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wvmfk" event={"ID":"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6","Type":"ContainerStarted","Data":"922edc89c34d0b3a7d09e86754085efe1fc6fb65be6bb773a380d35f5420af6d"}
Oct 30 00:34:41.033354 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:41.033308    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xfk62" podStartSLOduration=2.088940408 podStartE2EDuration="5.033296369s" totalImagesPullingTime="2.944355961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:41.032479091 +0000 UTC m=+45.905235333" watchObservedRunningTime="2026-10-30 00:34:41.033296369 +0000 UTC m=+45.906052591"
Oct 30 00:34:41.077539 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:41.077496    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wvmfk" podStartSLOduration=2.936384073 podStartE2EDuration="5.077482832s" totalImagesPullingTime="2.141098759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:34:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:41.076726471 +0000 UTC m=+45.949482694" watchObservedRunningTime="2026-10-30 00:34:41.077482832 +0000 UTC m=+45.950239054"
Oct 30 00:34:42.999896 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:42.999856    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj" event={"ID":"21be6874-6138-4a00-9c6e-aa1841eb3230","Type":"ContainerStarted","Data":"bd87a2995d63fb60f2310deca62ecab0f71ffeaa4d9a3799aba58ebda8b26950"}
Oct 30 00:34:43.000257 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:42.999918    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj" event={"ID":"21be6874-6138-4a00-9c6e-aa1841eb3230","Type":"ContainerStarted","Data":"67372056c3b02e80e80bcbc651b636e551dd58bf0ef23486de5a9101a75eac5e"}
Oct 30 00:34:43.043287 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:43.043246    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj" podStartSLOduration=1.827980187 podStartE2EDuration="3.043234694s" totalImagesPullingTime="1.215254507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:43.042575067 +0000 UTC m=+47.915331289" watchObservedRunningTime="2026-10-30 00:34:43.043234694 +0000 UTC m=+47.915990944"
Oct 30 00:34:44.734349 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.734317    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"]
Oct 30 00:34:44.738272 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.738241    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:34:44.761437 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.761417    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 00:34:44.762392 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.762376    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 00:34:44.775134 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.775114    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xjltb\""
Oct 30 00:34:44.776231 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.776208    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"]
Oct 30 00:34:44.776955 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.776930    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"]
Oct 30 00:34:44.778757 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.778742    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wb627"]
Oct 30 00:34:44.778876 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.778860    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.780448 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.780435    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.801286 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.801263    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 00:34:44.801286 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.801266    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 00:34:44.802372 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.802356    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"]
Oct 30 00:34:44.821938 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.821921    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 00:34:44.822103 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.822087    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 00:34:44.822229 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.822212    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 00:34:44.822289 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.822255    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mk2sw\""
Oct 30 00:34:44.822289 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.822266    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x9vrg\""
Oct 30 00:34:44.822356 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.822304    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 00:34:44.860132 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860110    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc8e4d97-4c7f-450d-961c-528c992a29b2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.860264 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/962a4e53-88b6-4866-abc5-108b6d5ce784-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:34:44.860264 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6gmn\" (UniqueName: \"kubernetes.io/projected/f53ec687-40dd-4358-bc05-83d064d9d082-kube-api-access-l6gmn\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.860264 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkp59\" (UniqueName: \"kubernetes.io/projected/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-api-access-mkp59\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.860264 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.860264 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-textfile\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.860524 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-tls\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.860524 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860410    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-accelerators-collector-config\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.860524 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860450    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.860524 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/962a4e53-88b6-4866-abc5-108b6d5ce784-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:34:44.860686 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860541    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f53ec687-40dd-4358-bc05-83d064d9d082-metrics-client-ca\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.860686 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-wtmp\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.860686 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/962a4e53-88b6-4866-abc5-108b6d5ce784-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:34:44.860686 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860649    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.860829 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860806    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhjkw\" (UniqueName: \"kubernetes.io/projected/962a4e53-88b6-4866-abc5-108b6d5ce784-kube-api-access-lhjkw\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:34:44.860872 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bc8e4d97-4c7f-450d-961c-528c992a29b2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.860909 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f53ec687-40dd-4358-bc05-83d064d9d082-sys\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.860909 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f53ec687-40dd-4358-bc05-83d064d9d082-root\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.860988 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.860964    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.961935 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.961906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f53ec687-40dd-4358-bc05-83d064d9d082-sys\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.961935 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.961937    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f53ec687-40dd-4358-bc05-83d064d9d082-root\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.962108 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.961973    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.962108 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962008    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f53ec687-40dd-4358-bc05-83d064d9d082-sys\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.962108 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962021    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f53ec687-40dd-4358-bc05-83d064d9d082-root\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.962108 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc8e4d97-4c7f-450d-961c-528c992a29b2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.962108 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/962a4e53-88b6-4866-abc5-108b6d5ce784-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:34:44.962108 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6gmn\" (UniqueName: \"kubernetes.io/projected/f53ec687-40dd-4358-bc05-83d064d9d082-kube-api-access-l6gmn\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.962446 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkp59\" (UniqueName: \"kubernetes.io/projected/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-api-access-mkp59\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.962446 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.962446 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-textfile\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.962446 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-tls\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.962446 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-accelerators-collector-config\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.962446 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.962446 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/962a4e53-88b6-4866-abc5-108b6d5ce784-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:34:44.962446 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f53ec687-40dd-4358-bc05-83d064d9d082-metrics-client-ca\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.962446 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-wtmp\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.962446 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/962a4e53-88b6-4866-abc5-108b6d5ce784-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:34:44.962446 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.962995 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhjkw\" (UniqueName: \"kubernetes.io/projected/962a4e53-88b6-4866-abc5-108b6d5ce784-kube-api-access-lhjkw\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:34:44.962995 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bc8e4d97-4c7f-450d-961c-528c992a29b2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.962995 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962860    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc8e4d97-4c7f-450d-961c-528c992a29b2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.962995 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.962877    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bc8e4d97-4c7f-450d-961c-528c992a29b2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.963200 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.963001    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/962a4e53-88b6-4866-abc5-108b6d5ce784-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:34:44.963200 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.963123    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-textfile\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.963488 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.963460    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.963594 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.963577    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-wtmp\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.963866 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.963843    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-accelerators-collector-config\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.964230 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.964207    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f53ec687-40dd-4358-bc05-83d064d9d082-metrics-client-ca\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.965500 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.965468    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.965500 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.965491    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/962a4e53-88b6-4866-abc5-108b6d5ce784-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:34:44.965653 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.965560    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.965754 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.965660    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/962a4e53-88b6-4866-abc5-108b6d5ce784-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:34:44.965754 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.965730    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-tls\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.965926 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.965906    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.982549 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.982529    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-57449f4d45-vtkst"]
Oct 30 00:34:44.985171 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.985137    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:44.994130 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.994110    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6gmn\" (UniqueName: \"kubernetes.io/projected/f53ec687-40dd-4358-bc05-83d064d9d082-kube-api-access-l6gmn\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:44.994321 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.994305    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkp59\" (UniqueName: \"kubernetes.io/projected/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-api-access-mkp59\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:44.996664 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:44.996642    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhjkw\" (UniqueName: \"kubernetes.io/projected/962a4e53-88b6-4866-abc5-108b6d5ce784-kube-api-access-lhjkw\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:34:45.006546 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.006391    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 00:34:45.006546 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.006391    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 00:34:45.006546 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.006429    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 00:34:45.006546 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.006469    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 00:34:45.006546 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.006495    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 00:34:45.006546 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.006535    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 00:34:45.006807 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.006547    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 00:34:45.006807 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.006614    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dff75\""
Oct 30 00:34:45.007458 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.007442    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57449f4d45-vtkst"]
Oct 30 00:34:45.051013 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.050996    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:34:45.062976 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.062957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.063064 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.063019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.063102 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.063077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.063135 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.063116    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-tmp\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.063172 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.063149    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.063221 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.063176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r59hp\" (UniqueName: \"kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.063297 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.063280    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.093564 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.093314    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:34:45.099980 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.099955    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:34:45.164201 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.164170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.164327 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.164205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59hp\" (UniqueName: \"kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.164327 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.164225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.164327 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.164271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.164327 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.164302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.164327 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.164315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.164577 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.164335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-tmp\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.164905 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.164634    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-tmp\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.165401 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.165384    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.165582 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.165564    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.166403 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.166380    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.167888 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.167871    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.167985 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.167968    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.193201 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.193178    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r59hp\" (UniqueName: \"kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.295320 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.295298    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:45.353466 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.353437    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"]
Oct 30 00:34:45.357800 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:34:45.357774    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod962a4e53_88b6_4866_abc5_108b6d5ce784.slice/crio-2595dae84d252abb34a540cda5b3a7def3276a7c59ef7bd7b8f625c216cc7a1c WatchSource:0}: Error finding container 2595dae84d252abb34a540cda5b3a7def3276a7c59ef7bd7b8f625c216cc7a1c: Status 404 returned error can't find the container with id 2595dae84d252abb34a540cda5b3a7def3276a7c59ef7bd7b8f625c216cc7a1c
Oct 30 00:34:45.393768 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.393747    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"]
Oct 30 00:34:45.395224 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:34:45.395196    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8e4d97_4c7f_450d_961c_528c992a29b2.slice/crio-a2f3cc0b7891f938dcbe4b74b6e5d04be4e232fd453c9d65e55304d591f42f56 WatchSource:0}: Error finding container a2f3cc0b7891f938dcbe4b74b6e5d04be4e232fd453c9d65e55304d591f42f56: Status 404 returned error can't find the container with id a2f3cc0b7891f938dcbe4b74b6e5d04be4e232fd453c9d65e55304d591f42f56
Oct 30 00:34:45.590437 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:45.590409    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57449f4d45-vtkst"]
Oct 30 00:34:45.590865 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:34:45.590834    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1fcdd5_b4bc_436f_9350_cfe138b3d9bd.slice/crio-4fa4df476257e406e2a2ef2135a8a7a2d2f203d6bbd313427afb78fe163973a0 WatchSource:0}: Error finding container 4fa4df476257e406e2a2ef2135a8a7a2d2f203d6bbd313427afb78fe163973a0: Status 404 returned error can't find the container with id 4fa4df476257e406e2a2ef2135a8a7a2d2f203d6bbd313427afb78fe163973a0
Oct 30 00:34:46.009404 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.009376    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57449f4d45-vtkst" event={"ID":"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd","Type":"ContainerStarted","Data":"4fa4df476257e406e2a2ef2135a8a7a2d2f203d6bbd313427afb78fe163973a0"}
Oct 30 00:34:46.010554 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.010524    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wb627" event={"ID":"f53ec687-40dd-4358-bc05-83d064d9d082","Type":"ContainerStarted","Data":"300b8d79f4ab4d8f825f4849a4f77cee8659de361bab4210aaca9b8fc3e409e8"}
Oct 30 00:34:46.011605 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.011584    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz" event={"ID":"bc8e4d97-4c7f-450d-961c-528c992a29b2","Type":"ContainerStarted","Data":"a2f3cc0b7891f938dcbe4b74b6e5d04be4e232fd453c9d65e55304d591f42f56"}
Oct 30 00:34:46.013380 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.013361    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh" event={"ID":"962a4e53-88b6-4866-abc5-108b6d5ce784","Type":"ContainerStarted","Data":"70c8d7afd2d665f0e975bb3463d320aabe3453719590c5a4f52266a1b7eb263f"}
Oct 30 00:34:46.013460 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.013400    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh" event={"ID":"962a4e53-88b6-4866-abc5-108b6d5ce784","Type":"ContainerStarted","Data":"71b0625d7c49f5e7c132a6f9a13d6b6d621c4e57206c8a275d31aae5d1c0ae8d"}
Oct 30 00:34:46.013460 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.013418    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh" event={"ID":"962a4e53-88b6-4866-abc5-108b6d5ce784","Type":"ContainerStarted","Data":"2595dae84d252abb34a540cda5b3a7def3276a7c59ef7bd7b8f625c216cc7a1c"}
Oct 30 00:34:46.227564 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.226001    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 00:34:46.230354 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.229275    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.251288 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.251264    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 00:34:46.251407 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.251359    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 00:34:46.252223 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.252201    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 00:34:46.252373 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.252278    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 00:34:46.252373 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.252310    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 00:34:46.252373 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.252338    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 00:34:46.252373 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.252343    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-82whn\""
Oct 30 00:34:46.252599 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.252395    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 00:34:46.252599 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.252395    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 00:34:46.274618 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.274586    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.274768 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.274646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.274768 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.274677    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-web-config\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.274768 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.274702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.274768 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.274751    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.274982 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.274775    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.274982 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.274807    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.274982 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.274839    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-out\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.274982 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.274871    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.274982 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.274897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwm8g\" (UniqueName: \"kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-kube-api-access-fwm8g\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.274982 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.274929    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.274982 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.274955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.275307 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.275027    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.279590 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.279546    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 00:34:46.281096 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.281072    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 00:34:46.376847 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.376081    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.376847 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.376122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.376847 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.376163    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.376847 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.376196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-web-config\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.376847 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.376214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.376847 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.376235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.376847 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.376259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.376847 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.376290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.376847 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.376310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-out\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.376847 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.376331    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.376847 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.376366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwm8g\" (UniqueName: \"kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-kube-api-access-fwm8g\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.376847 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.376408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.376847 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.376427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.380050 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.379403    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.381352 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.380655    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.381352 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.381009    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.381987 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.381962    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.383507 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.382755    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.383507 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.382925    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-out\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.383507 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.383221    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.383722 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.383668    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.384386 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.384328    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.384479 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.384456    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-web-config\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.385449 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.385429    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.385777 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.385761    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.405510 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.405462    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwm8g\" (UniqueName: \"kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-kube-api-access-fwm8g\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.553265 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.553225    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:34:46.894237 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:46.894210    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 00:34:46.924009 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:34:46.923978    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426a04b3_dc76_4b89_8d28_3038652fd7bf.slice/crio-00ed8e3406e02fbd84b919cd0d099d011999035f9933db5cc62581e6107c63a1 WatchSource:0}: Error finding container 00ed8e3406e02fbd84b919cd0d099d011999035f9933db5cc62581e6107c63a1: Status 404 returned error can't find the container with id 00ed8e3406e02fbd84b919cd0d099d011999035f9933db5cc62581e6107c63a1
Oct 30 00:34:47.017797 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.017766    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"426a04b3-dc76-4b89-8d28-3038652fd7bf","Type":"ContainerStarted","Data":"00ed8e3406e02fbd84b919cd0d099d011999035f9933db5cc62581e6107c63a1"}
Oct 30 00:34:47.019454 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.019431    2581 generic.go:356] "Generic (PLEG): container finished" podID="f53ec687-40dd-4358-bc05-83d064d9d082" containerID="742019ae4ad77df39ea4746b0a561d5778e19c03fb8a5df83a90532a0c62b4ef" exitCode=0
Oct 30 00:34:47.019523 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.019498    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wb627" event={"ID":"f53ec687-40dd-4358-bc05-83d064d9d082","Type":"ContainerDied","Data":"742019ae4ad77df39ea4746b0a561d5778e19c03fb8a5df83a90532a0c62b4ef"}
Oct 30 00:34:47.022331 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.022303    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz" event={"ID":"bc8e4d97-4c7f-450d-961c-528c992a29b2","Type":"ContainerStarted","Data":"61e85537db613c788a3a3e2e5fd0699fcb2ed89bd9cef8bd15dd5fe0ea5547a4"}
Oct 30 00:34:47.022402 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.022359    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz" event={"ID":"bc8e4d97-4c7f-450d-961c-528c992a29b2","Type":"ContainerStarted","Data":"c40aa4c7fcd55749bf1fb630c9942e6230c0991ee34ca8ce647e5f6ac80aaff2"}
Oct 30 00:34:47.027621 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.027603    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh" event={"ID":"962a4e53-88b6-4866-abc5-108b6d5ce784","Type":"ContainerStarted","Data":"4e8d9ddb19f4cca85e1e54ee4a93b4fbe323d4472646be92aeee3f95e1919a08"}
Oct 30 00:34:47.035919 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.035899    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-676949fd44-65zsj"]
Oct 30 00:34:47.042144 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.042128    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.064351 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.061981    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 00:34:47.064351 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.062048    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 00:34:47.064351 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.062299    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 00:34:47.064351 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.062363    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 00:34:47.064351 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.062460    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9qdg95h0bhori\""
Oct 30 00:34:47.064351 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.062509    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 00:34:47.064351 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.064019    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-676949fd44-65zsj"]
Oct 30 00:34:47.064351 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.064145    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xzvlc\""
Oct 30 00:34:47.085301 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.085236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.085440 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.085305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-tls\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.085495 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.085478    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-grpc-tls\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.085527 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.085511    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpw5w\" (UniqueName: \"kubernetes.io/projected/c5282344-9beb-485c-9525-6b09bdda0804-kube-api-access-wpw5w\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.085594 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.085580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.085628 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.085613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5282344-9beb-485c-9525-6b09bdda0804-tmp\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.085674 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.085660    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5282344-9beb-485c-9525-6b09bdda0804-metrics-client-ca\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.085732 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.085699    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.085808 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.085776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.167671 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.166623    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh" podStartSLOduration=2.010199333 podStartE2EDuration="3.166607151s" totalImagesPullingTime="1.156407818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:47.166421226 +0000 UTC m=+52.039177450" watchObservedRunningTime="2026-10-30 00:34:47.166607151 +0000 UTC m=+52.039363376"
Oct 30 00:34:47.187176 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.187148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.187294 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.187190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5282344-9beb-485c-9525-6b09bdda0804-tmp\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.187294 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.187218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5282344-9beb-485c-9525-6b09bdda0804-metrics-client-ca\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.187294 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.187247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.187294 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.187284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.187493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.187316    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.187493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.187349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-tls\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.187493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.187416    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-grpc-tls\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.187493 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.187442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpw5w\" (UniqueName: \"kubernetes.io/projected/c5282344-9beb-485c-9525-6b09bdda0804-kube-api-access-wpw5w\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.188345 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.188281    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5282344-9beb-485c-9525-6b09bdda0804-tmp\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.188873 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.188845    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5282344-9beb-485c-9525-6b09bdda0804-metrics-client-ca\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.191852 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.191823    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.191931 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.191877    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-tls\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.191931 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.191917    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-grpc-tls\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.193383 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.193340    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.193536 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.193398    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.195812 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.195332    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.214907 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.214878    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpw5w\" (UniqueName: \"kubernetes.io/projected/c5282344-9beb-485c-9525-6b09bdda0804-kube-api-access-wpw5w\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.393773 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.392840    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:47.718959 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:47.718929    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-676949fd44-65zsj"]
Oct 30 00:34:48.030755 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:48.030695    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-676949fd44-65zsj" event={"ID":"c5282344-9beb-485c-9525-6b09bdda0804","Type":"ContainerStarted","Data":"bc17043616d9ef84669e5f37b1297765b0dfc2cc919ce4683ea5831135982716"}
Oct 30 00:34:48.033562 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:48.033531    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wb627" event={"ID":"f53ec687-40dd-4358-bc05-83d064d9d082","Type":"ContainerStarted","Data":"438a51181370ad8827d70ae3c8c6b76f579d8b388b845336fcfa4275ecd8f777"}
Oct 30 00:34:48.033671 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:48.033583    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wb627" event={"ID":"f53ec687-40dd-4358-bc05-83d064d9d082","Type":"ContainerStarted","Data":"049f21434701fa9f50f5e043c7642dedc0b096ccc0a03e922f9c51db263ab41e"}
Oct 30 00:34:48.036048 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:48.036021    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz" event={"ID":"bc8e4d97-4c7f-450d-961c-528c992a29b2","Type":"ContainerStarted","Data":"98f4f4e1494404ca64d088b2113608b3a9dc9b178188183acd76f3967f734507"}
Oct 30 00:34:48.037694 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:48.037670    2581 generic.go:356] "Generic (PLEG): container finished" podID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerID="5be1dff3726a9bbaf208d7e7c745e2af4e1fb89d65e25c3dfb3564f5a9011269" exitCode=0
Oct 30 00:34:48.037807 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:48.037780    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"426a04b3-dc76-4b89-8d28-3038652fd7bf","Type":"ContainerDied","Data":"5be1dff3726a9bbaf208d7e7c745e2af4e1fb89d65e25c3dfb3564f5a9011269"}
Oct 30 00:34:48.079715 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:48.079669    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wb627" podStartSLOduration=3.13953742 podStartE2EDuration="4.079654166s" totalImagesPullingTime="853.986994ms" totalInitContainerRuntime="86.129752ms" isStatefulPod=true podCreationTimestamp="2026-10-30 00:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:48.079061811 +0000 UTC m=+52.951818035" watchObservedRunningTime="2026-10-30 00:34:48.079654166 +0000 UTC m=+52.952410396"
Oct 30 00:34:48.125956 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:48.125905    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz" podStartSLOduration=2.687464508 podStartE2EDuration="4.125890657s" totalImagesPullingTime="1.438426149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:48.124959423 +0000 UTC m=+52.997715634" watchObservedRunningTime="2026-10-30 00:34:48.125890657 +0000 UTC m=+52.998646880"
Oct 30 00:34:49.042474 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.042434    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57449f4d45-vtkst" event={"ID":"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd","Type":"ContainerStarted","Data":"267053f4f6c5e5cb54f2f2febf56a0d4e71d0fde1a936ac35691f8033722299e"}
Oct 30 00:34:49.088075 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.088017    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-57449f4d45-vtkst" podStartSLOduration=1.8199017469999998 podStartE2EDuration="5.087999801s" totalImagesPullingTime="3.268098054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:49.087115749 +0000 UTC m=+53.959871986" watchObservedRunningTime="2026-10-30 00:34:49.087999801 +0000 UTC m=+53.960756028"
Oct 30 00:34:49.288911 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.288830    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7c55cf7f78-h65h8"]
Oct 30 00:34:49.291200 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.291175    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.307850 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.307824    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 00:34:49.308036 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.307867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b8c48d07-5599-4a74-94d4-d6488bde2827-client-ca-bundle\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.308036 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.307984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c48d07-5599-4a74-94d4-d6488bde2827-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.308036 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.308029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b8c48d07-5599-4a74-94d4-d6488bde2827-secret-metrics-server-client-certs\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.308178 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.308047    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 00:34:49.308178 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.308056    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b8c48d07-5599-4a74-94d4-d6488bde2827-metrics-server-audit-profiles\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.308178 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.308060    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-78e6gfjg9uta9\""
Oct 30 00:34:49.308178 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.308117    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 00:34:49.308178 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.308117    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ks6f2\""
Oct 30 00:34:49.308379 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.308198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b8c48d07-5599-4a74-94d4-d6488bde2827-audit-log\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.308379 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.308234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmnmx\" (UniqueName: \"kubernetes.io/projected/b8c48d07-5599-4a74-94d4-d6488bde2827-kube-api-access-qmnmx\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.308379 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.308236    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 00:34:49.308379 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.308268    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b8c48d07-5599-4a74-94d4-d6488bde2827-secret-metrics-server-tls\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.322809 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.322774    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7c55cf7f78-h65h8"]
Oct 30 00:34:49.409126 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.409084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b8c48d07-5599-4a74-94d4-d6488bde2827-client-ca-bundle\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.409312 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.409169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c48d07-5599-4a74-94d4-d6488bde2827-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.409312 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.409216    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b8c48d07-5599-4a74-94d4-d6488bde2827-secret-metrics-server-client-certs\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.409312 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.409246    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b8c48d07-5599-4a74-94d4-d6488bde2827-metrics-server-audit-profiles\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.409312 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.409305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b8c48d07-5599-4a74-94d4-d6488bde2827-audit-log\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.409491 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.409339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmnmx\" (UniqueName: \"kubernetes.io/projected/b8c48d07-5599-4a74-94d4-d6488bde2827-kube-api-access-qmnmx\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.409491 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.409365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b8c48d07-5599-4a74-94d4-d6488bde2827-secret-metrics-server-tls\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.410239 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.410210    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b8c48d07-5599-4a74-94d4-d6488bde2827-audit-log\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.410390 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.410367    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c48d07-5599-4a74-94d4-d6488bde2827-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.410592 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.410565    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b8c48d07-5599-4a74-94d4-d6488bde2827-metrics-server-audit-profiles\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.412098 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.412074    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b8c48d07-5599-4a74-94d4-d6488bde2827-client-ca-bundle\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.412883 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.412861    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b8c48d07-5599-4a74-94d4-d6488bde2827-secret-metrics-server-tls\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.416391 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.416367    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b8c48d07-5599-4a74-94d4-d6488bde2827-secret-metrics-server-client-certs\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.437297 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.437275    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmnmx\" (UniqueName: \"kubernetes.io/projected/b8c48d07-5599-4a74-94d4-d6488bde2827-kube-api-access-qmnmx\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.603242 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.603060    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:34:49.641085 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.641048    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-tljgj"]
Oct 30 00:34:49.645728 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.645460    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj"
Oct 30 00:34:49.665065 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.665011    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-x2hxt\""
Oct 30 00:34:49.670864 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.665393    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 00:34:49.674127 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.674064    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-tljgj"]
Oct 30 00:34:49.713376 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.713348    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12bf7b6a-fd62-4921-910d-f9fff2b347d1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-tljgj\" (UID: \"12bf7b6a-fd62-4921-910d-f9fff2b347d1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj"
Oct 30 00:34:49.713469 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.713415    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7spz\" (UniqueName: \"kubernetes.io/projected/12bf7b6a-fd62-4921-910d-f9fff2b347d1-kube-api-access-j7spz\") pod \"monitoring-plugin-77f4f76594-tljgj\" (UID: \"12bf7b6a-fd62-4921-910d-f9fff2b347d1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj"
Oct 30 00:34:49.814785 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.814759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12bf7b6a-fd62-4921-910d-f9fff2b347d1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-tljgj\" (UID: \"12bf7b6a-fd62-4921-910d-f9fff2b347d1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj"
Oct 30 00:34:49.814876 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.814809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7spz\" (UniqueName: \"kubernetes.io/projected/12bf7b6a-fd62-4921-910d-f9fff2b347d1-kube-api-access-j7spz\") pod \"monitoring-plugin-77f4f76594-tljgj\" (UID: \"12bf7b6a-fd62-4921-910d-f9fff2b347d1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj"
Oct 30 00:34:49.817259 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.817231    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12bf7b6a-fd62-4921-910d-f9fff2b347d1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-tljgj\" (UID: \"12bf7b6a-fd62-4921-910d-f9fff2b347d1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj"
Oct 30 00:34:49.840630 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.840603    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7spz\" (UniqueName: \"kubernetes.io/projected/12bf7b6a-fd62-4921-910d-f9fff2b347d1-kube-api-access-j7spz\") pod \"monitoring-plugin-77f4f76594-tljgj\" (UID: \"12bf7b6a-fd62-4921-910d-f9fff2b347d1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj"
Oct 30 00:34:49.930923 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.930894    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7c55cf7f78-h65h8"]
Oct 30 00:34:49.933370 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:34:49.933347    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c48d07_5599_4a74_94d4_d6488bde2827.slice/crio-7bab66348e408303b7eb14327330fe90dc78ec9222ea1bf64848cc0d9b162120 WatchSource:0}: Error finding container 7bab66348e408303b7eb14327330fe90dc78ec9222ea1bf64848cc0d9b162120: Status 404 returned error can't find the container with id 7bab66348e408303b7eb14327330fe90dc78ec9222ea1bf64848cc0d9b162120
Oct 30 00:34:49.946135 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.946115    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:34:49.992823 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:49.992798    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj"
Oct 30 00:34:50.069804 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.069674    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-676949fd44-65zsj" event={"ID":"c5282344-9beb-485c-9525-6b09bdda0804","Type":"ContainerStarted","Data":"0fdf612a0f6bf98b89257ee5bb3d9b467214bc6ba59d664f56f13175e5a7d1b0"}
Oct 30 00:34:50.071556 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.071483    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8" event={"ID":"b8c48d07-5599-4a74-94d4-d6488bde2827","Type":"ContainerStarted","Data":"7bab66348e408303b7eb14327330fe90dc78ec9222ea1bf64848cc0d9b162120"}
Oct 30 00:34:50.078419 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.078141    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"426a04b3-dc76-4b89-8d28-3038652fd7bf","Type":"ContainerStarted","Data":"56e4c08f9e42d3ea818b5e4c8fd1bcf4ed93ca026c3e47fb577c6e0a6e801432"}
Oct 30 00:34:50.078419 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.078204    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"426a04b3-dc76-4b89-8d28-3038652fd7bf","Type":"ContainerStarted","Data":"5515e80e8e28805e47ca2d5dd4127719c0b4fd426c264e0ec0becc2f84687f9d"}
Oct 30 00:34:50.078419 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.078242    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"426a04b3-dc76-4b89-8d28-3038652fd7bf","Type":"ContainerStarted","Data":"bf396d37b29bfa2006e3935fa69145a8acb684784ba731ede764600f0f477060"}
Oct 30 00:34:50.078419 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.078278    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"426a04b3-dc76-4b89-8d28-3038652fd7bf","Type":"ContainerStarted","Data":"6a69994adc0ccc20c20c18cc3c4b2db5e34a2175084c002010effc404a0b33b6"}
Oct 30 00:34:50.078419 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.078315    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"426a04b3-dc76-4b89-8d28-3038652fd7bf","Type":"ContainerStarted","Data":"273b61e0df7f8e20c3a0d13e72a028b7f29b04f04826d53026d3244a3444304f"}
Oct 30 00:34:50.087747 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.087725    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6ddf586968-ml4vv"]
Oct 30 00:34:50.095422 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.095377    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.114874 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.114854    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6ddf586968-ml4vv"]
Oct 30 00:34:50.116690 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.116665    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 00:34:50.116805 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.116784    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 00:34:50.116861 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.116839    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 00:34:50.116917 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.116871    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 00:34:50.116917 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.116878    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 00:34:50.117020 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.116972    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-r25c7\""
Oct 30 00:34:50.117880 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.117862    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-federate-client-tls\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.117955 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.117892    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ncsp\" (UniqueName: \"kubernetes.io/projected/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-kube-api-access-9ncsp\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.117955 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.117921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.118048 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.117981    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-telemeter-client-tls\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.118048 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.118007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-metrics-client-ca\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.118150 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.118094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-secret-telemeter-client\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.120796 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.120768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.120884 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.120840    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-serving-certs-ca-bundle\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.130543 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.130509    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 00:34:50.221803 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.221769    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-serving-certs-ca-bundle\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.221941 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.221838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-federate-client-tls\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.221941 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.221857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ncsp\" (UniqueName: \"kubernetes.io/projected/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-kube-api-access-9ncsp\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.221941 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.221875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.221941 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.221899    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-telemeter-client-tls\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.221941 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.221913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-metrics-client-ca\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.222193 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.221951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-secret-telemeter-client\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.222193 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.221982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.222623 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.222594    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-serving-certs-ca-bundle\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.222828 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.222804    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.223039 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.223013    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-metrics-client-ca\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.224355 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.224330    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-federate-client-tls\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.224461 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.224383    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.224461 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.224413    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-secret-telemeter-client\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.224461 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.224454    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-telemeter-client-tls\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.246186 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.246161    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ncsp\" (UniqueName: \"kubernetes.io/projected/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-kube-api-access-9ncsp\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.304368 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.304346    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-tljgj"]
Oct 30 00:34:50.304929 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:34:50.304908    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12bf7b6a_fd62_4921_910d_f9fff2b347d1.slice/crio-010e66bf2b3c5d50da7b1121c4fc2b867ec2bd71013584ba7dd0c3c6dd07b779 WatchSource:0}: Error finding container 010e66bf2b3c5d50da7b1121c4fc2b867ec2bd71013584ba7dd0c3c6dd07b779: Status 404 returned error can't find the container with id 010e66bf2b3c5d50da7b1121c4fc2b867ec2bd71013584ba7dd0c3c6dd07b779
Oct 30 00:34:50.415131 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.415108    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:34:50.734461 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.734393    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6ddf586968-ml4vv"]
Oct 30 00:34:50.742956 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:34:50.742925    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb2dce5_50be_42dc_88f4_ecb4d6a29d28.slice/crio-bbdeeec904dc0720fe0f45061c51cf73759829349eb4521341d423a42cf61960 WatchSource:0}: Error finding container bbdeeec904dc0720fe0f45061c51cf73759829349eb4521341d423a42cf61960: Status 404 returned error can't find the container with id bbdeeec904dc0720fe0f45061c51cf73759829349eb4521341d423a42cf61960
Oct 30 00:34:50.997617 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:50.997597    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:34:51.087463 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.087439    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj" event={"ID":"12bf7b6a-fd62-4921-910d-f9fff2b347d1","Type":"ContainerStarted","Data":"010e66bf2b3c5d50da7b1121c4fc2b867ec2bd71013584ba7dd0c3c6dd07b779"}
Oct 30 00:34:51.093729 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.093688    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"426a04b3-dc76-4b89-8d28-3038652fd7bf","Type":"ContainerStarted","Data":"23deb9e025e27d9e7dbb292cd389ecc5fcb9c5371ebe72e02ecfabdcbde56444"}
Oct 30 00:34:51.100316 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.100065    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-676949fd44-65zsj" event={"ID":"c5282344-9beb-485c-9525-6b09bdda0804","Type":"ContainerStarted","Data":"933625b4a2a195bc9d8376e11423f443d101f4bf0413881097745bade324e097"}
Oct 30 00:34:51.100316 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.100137    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-676949fd44-65zsj" event={"ID":"c5282344-9beb-485c-9525-6b09bdda0804","Type":"ContainerStarted","Data":"1884118f8f8e42cccc988e56d1e73768bd256f18e3d3913fe24dc07c4b2a400b"}
Oct 30 00:34:51.100316 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.100218    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-676949fd44-65zsj" event={"ID":"c5282344-9beb-485c-9525-6b09bdda0804","Type":"ContainerStarted","Data":"aab2e0c195ac8fb336a53173bce2de0dc9aa77340edad074a09105b7871ea5cb"}
Oct 30 00:34:51.100316 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.100261    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-676949fd44-65zsj" event={"ID":"c5282344-9beb-485c-9525-6b09bdda0804","Type":"ContainerStarted","Data":"651c7d1fe57565ca3aa2b276437702e918319af336282b963ce1fa14e64c4c39"}
Oct 30 00:34:51.104321 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.104294    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv" event={"ID":"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28","Type":"ContainerStarted","Data":"bbdeeec904dc0720fe0f45061c51cf73759829349eb4521341d423a42cf61960"}
Oct 30 00:34:51.151008 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.150968    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.720838136 podStartE2EDuration="5.150955342s" totalImagesPullingTime="3.409974424s" totalInitContainerRuntime="20.142782ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:51.150368394 +0000 UTC m=+56.023124642" watchObservedRunningTime="2026-10-30 00:34:51.150955342 +0000 UTC m=+56.023711564"
Oct 30 00:34:51.729738 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.729133    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:34:51.732486 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.732448    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.749328 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.749304    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 00:34:51.749586 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.749571    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 00:34:51.749682 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.749669    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 00:34:51.749870 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.749854    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 00:34:51.749935 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.749923    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 00:34:51.750163 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.750149    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 00:34:51.762676 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.762654    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hxlc4\""
Oct 30 00:34:51.762822 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.762723    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 00:34:51.763702 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.763606    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 00:34:51.764473 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.764456    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 00:34:51.777868 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.776497    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7f0anuf4jt4pi\""
Oct 30 00:34:51.777868 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.776583    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 00:34:51.777868 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.776682    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 00:34:51.777868 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.776873    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 00:34:51.788731 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.785137    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 00:34:51.792278 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.792205    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:34:51.794429 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.794408    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 00:34:51.839255 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5kb7\" (UniqueName: \"kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-kube-api-access-j5kb7\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.839406 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-config-out\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.839406 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.839406 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.839406 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.839406 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.839664 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839444    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.839664 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.839664 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.839664 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839548    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.839664 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.839664 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839608    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.839664 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.839664 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839652    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.840044 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.840044 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.840044 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.840044 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839825    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-config\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.840044 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.840044 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.839869    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-web-config\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.940814 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.940742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.940814 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.940792    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.941034 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.940817    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.941034 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.940835    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.941034 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.940864    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.941034 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.940898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.941034 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.940917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.941034 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.940968    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-config\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.941034 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.940991    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.941034 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.941015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-web-config\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.941431 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.941047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5kb7\" (UniqueName: \"kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-kube-api-access-j5kb7\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.941431 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.941084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-config-out\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.941431 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.941106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.941431 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.941137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.941431 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.941166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.941431 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.941195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.941431 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.941259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.941431 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.941318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.941431 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.941357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.941431 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.941383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.942301 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.942272    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.946141 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.944300    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.946141 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.944961    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.946141 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.945243    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.946141 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.945828    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.946141 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.945972    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.946986 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.946594    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.946986 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.946690    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.946986 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.946867    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.947490 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.947191    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.948136 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.947864    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.948136 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.948090    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-config-out\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.948496 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.948470    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.950042 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.950018    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-config\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.950659 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.950621    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.958717 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.958682    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-web-config\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.960630 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.960603    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.960852 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.960795    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.961465 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.961443    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:51.967447 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:51.967426    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5kb7\" (UniqueName: \"kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-kube-api-access-j5kb7\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:52.066777 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:52.066741    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:34:52.111969 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:52.111935    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8" event={"ID":"b8c48d07-5599-4a74-94d4-d6488bde2827","Type":"ContainerStarted","Data":"fae453e0421227953517bd4cbdea796e3b16dd527ebad879b8f7c2fa8382643b"}
Oct 30 00:34:52.125266 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:52.125236    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-676949fd44-65zsj" event={"ID":"c5282344-9beb-485c-9525-6b09bdda0804","Type":"ContainerStarted","Data":"ad39f55cdaf4d2cefcec5885278cf067a4597ed93b2899bbc1afa5e5c0ac830e"}
Oct 30 00:34:52.125500 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:52.125310    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:34:52.158736 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:52.158668    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8" podStartSLOduration=1.472183345 podStartE2EDuration="3.158649488s" totalImagesPullingTime="1.686466143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:52.156685556 +0000 UTC m=+57.029441782" watchObservedRunningTime="2026-10-30 00:34:52.158649488 +0000 UTC m=+57.031405712"
Oct 30 00:34:52.211289 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:52.211201    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-676949fd44-65zsj" podStartSLOduration=2.187901735 podStartE2EDuration="5.211185444s" totalImagesPullingTime="3.023283709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:52.209856001 +0000 UTC m=+57.082612222" watchObservedRunningTime="2026-10-30 00:34:52.211185444 +0000 UTC m=+57.083941667"
Oct 30 00:34:52.405885 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:52.405850    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:34:52.406490 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:34:52.406467    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026218e5_d0ac_48b3_9023_f245d5cf9848.slice/crio-d9ad9b60e9424a59c50647f75bc4c95b224255b4e983bf3a664f5b06e50e507c WatchSource:0}: Error finding container d9ad9b60e9424a59c50647f75bc4c95b224255b4e983bf3a664f5b06e50e507c: Status 404 returned error can't find the container with id d9ad9b60e9424a59c50647f75bc4c95b224255b4e983bf3a664f5b06e50e507c
Oct 30 00:34:53.129063 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:53.129032    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv" event={"ID":"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28","Type":"ContainerStarted","Data":"a80c4da05ae62515ec6cb4adb44b45af19c14d99f9295a962269d9ddde2ddb5b"}
Oct 30 00:34:53.129494 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:53.129093    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv" event={"ID":"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28","Type":"ContainerStarted","Data":"6b5ae473fbd2510a80dd222d973e382cf57de57407d6005d0470b3b09a4128f0"}
Oct 30 00:34:53.129494 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:53.129121    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv" event={"ID":"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28","Type":"ContainerStarted","Data":"8c4a5a64157a24a37098aa82bfd163fd3b2db43b213e035093e67fde8d822ff9"}
Oct 30 00:34:53.130603 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:53.130580    2581 generic.go:356] "Generic (PLEG): container finished" podID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerID="289fa18bc40491e3b05eba723fa2e20720972feb42138c2f8902d6a3f206dc45" exitCode=0
Oct 30 00:34:53.130729 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:53.130672    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026218e5-d0ac-48b3-9023-f245d5cf9848","Type":"ContainerDied","Data":"289fa18bc40491e3b05eba723fa2e20720972feb42138c2f8902d6a3f206dc45"}
Oct 30 00:34:53.130798 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:53.130766    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026218e5-d0ac-48b3-9023-f245d5cf9848","Type":"ContainerStarted","Data":"d9ad9b60e9424a59c50647f75bc4c95b224255b4e983bf3a664f5b06e50e507c"}
Oct 30 00:34:53.137003 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:53.136975    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj" event={"ID":"12bf7b6a-fd62-4921-910d-f9fff2b347d1","Type":"ContainerStarted","Data":"af754ac67bafd36545ef8729a45da7a914d0ef91cdc1dc69971ccb7418bac715"}
Oct 30 00:34:53.137185 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:53.137161    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj"
Oct 30 00:34:53.142251 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:53.142230    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj"
Oct 30 00:34:53.178597 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:53.178517    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv" podStartSLOduration=1.261432292 podStartE2EDuration="3.178506173s" totalImagesPullingTime="1.917073881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:53.177344489 +0000 UTC m=+58.050100712" watchObservedRunningTime="2026-10-30 00:34:53.178506173 +0000 UTC m=+58.051262394"
Oct 30 00:34:53.277140 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:53.277099    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj" podStartSLOduration=2.504069982 podStartE2EDuration="4.277086939s" totalImagesPullingTime="1.773016957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:53.275910563 +0000 UTC m=+58.148666786" watchObservedRunningTime="2026-10-30 00:34:53.277086939 +0000 UTC m=+58.149843162"
Oct 30 00:34:55.295576 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:55.295540    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:55.296160 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:55.295671    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:55.300798 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:55.300774    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:56.153808 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:56.153780    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:34:57.156888 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:57.156848    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026218e5-d0ac-48b3-9023-f245d5cf9848","Type":"ContainerStarted","Data":"bbf9954a9960660aab5cc5a2c2c53899fcc2550f9e97de9917b1ddbe65246f62"}
Oct 30 00:34:57.157269 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:57.156926    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026218e5-d0ac-48b3-9023-f245d5cf9848","Type":"ContainerStarted","Data":"3f5f67376608b0f94747220f6a0557fe980a6367187405145ca6b992c1b0bf4c"}
Oct 30 00:34:57.157269 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:57.156974    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026218e5-d0ac-48b3-9023-f245d5cf9848","Type":"ContainerStarted","Data":"0c22a82ec34c5be2bdb4330d03eafa5e2924a2600f436618fed37ee98885eab8"}
Oct 30 00:34:57.157269 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:57.157012    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026218e5-d0ac-48b3-9023-f245d5cf9848","Type":"ContainerStarted","Data":"4000241637e9c13b529d8b826baffde60b8ede2175e48dcb09f5941ac1fd7e42"}
Oct 30 00:34:57.157269 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:57.157048    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026218e5-d0ac-48b3-9023-f245d5cf9848","Type":"ContainerStarted","Data":"061004aa234d1293a1a255e1c7f07cbba3e09f44d4799f45a066ca1b18ef5aa3"}
Oct 30 00:34:57.157269 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:57.157082    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026218e5-d0ac-48b3-9023-f245d5cf9848","Type":"ContainerStarted","Data":"4e74e8f29a1a34981cd8627fe553677682337f9869b8d7ed78dccf8dc2ef7271"}
Oct 30 00:34:57.216627 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:57.216581    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.032540434 podStartE2EDuration="6.216568972s" totalImagesPullingTime="3.147534033s" totalInitContainerRuntime="36.494505ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:34:57.216081024 +0000 UTC m=+62.088837250" watchObservedRunningTime="2026-10-30 00:34:57.216568972 +0000 UTC m=+62.089325194"
Oct 30 00:34:58.143792 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:34:58.143762    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:00.536308 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:00.536277    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:35:00.536747 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:00.536316    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhmk\" (UniqueName: \"kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk\") pod \"network-check-target-62gpq\" (UID: \"c9803898-12ec-4a07-9a8f-1eef3f89235b\") " pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:35:00.536747 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:00.536335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs\") pod \"network-metrics-daemon-wb6st\" (UID: \"3a145ece-ad48-43c2-a259-b9590802cfe8\") " pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:35:00.565628 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:00.565599    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 00:35:00.565774 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:00.565687    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 00:35:00.565774 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:00.565725    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 00:35:00.569050 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:00.569028    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:35:00.569197 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:00.569180    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs\") pod \"network-metrics-daemon-wb6st\" (UID: \"3a145ece-ad48-43c2-a259-b9590802cfe8\") " pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:35:00.581700 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:00.581679    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 00:35:00.590357 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:00.590334    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbhmk\" (UniqueName: \"kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk\") pod \"network-check-target-62gpq\" (UID: \"c9803898-12ec-4a07-9a8f-1eef3f89235b\") " pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:35:00.835131 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:00.835100    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-78zxw\""
Oct 30 00:35:00.840295 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:00.840275    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:35:00.849439 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:00.849316    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q972j\""
Oct 30 00:35:00.849543 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:00.849467    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-smldl\""
Oct 30 00:35:00.854274 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:00.854250    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:35:00.859468 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:00.859445    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:35:01.147063 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:01.147037    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-62gpq"]
Oct 30 00:35:01.147182 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:35:01.147038    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9803898_12ec_4a07_9a8f_1eef3f89235b.slice/crio-9070bded93cff1254d1950c7952ed0da996728ddbbbf7b7aef1623fbe8630bc8 WatchSource:0}: Error finding container 9070bded93cff1254d1950c7952ed0da996728ddbbbf7b7aef1623fbe8630bc8: Status 404 returned error can't find the container with id 9070bded93cff1254d1950c7952ed0da996728ddbbbf7b7aef1623fbe8630bc8
Oct 30 00:35:01.172033 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:01.172003    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-62gpq" event={"ID":"c9803898-12ec-4a07-9a8f-1eef3f89235b","Type":"ContainerStarted","Data":"9070bded93cff1254d1950c7952ed0da996728ddbbbf7b7aef1623fbe8630bc8"}
Oct 30 00:35:01.174618 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:01.174595    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7vsmg"]
Oct 30 00:35:01.175135 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:35:01.175113    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f00b4a_044e_49ab_8f43_4351590692e8.slice/crio-53f1621289f5ad3c9fcaeae350740d13f3f6b7cb39593a65dc2264873ea6bd83 WatchSource:0}: Error finding container 53f1621289f5ad3c9fcaeae350740d13f3f6b7cb39593a65dc2264873ea6bd83: Status 404 returned error can't find the container with id 53f1621289f5ad3c9fcaeae350740d13f3f6b7cb39593a65dc2264873ea6bd83
Oct 30 00:35:01.186471 ip-10-0-130-245 kubenswrapper[2581]: W1030 00:35:01.186444    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a145ece_ad48_43c2_a259_b9590802cfe8.slice/crio-81b266645f022c93bcaa424f1f6a9913bb476b091185d83906084d744c73279c WatchSource:0}: Error finding container 81b266645f022c93bcaa424f1f6a9913bb476b091185d83906084d744c73279c: Status 404 returned error can't find the container with id 81b266645f022c93bcaa424f1f6a9913bb476b091185d83906084d744c73279c
Oct 30 00:35:01.186562 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:01.186470    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wb6st"]
Oct 30 00:35:02.067043 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:02.067012    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:02.179120 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:02.179082    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wb6st" event={"ID":"3a145ece-ad48-43c2-a259-b9590802cfe8","Type":"ContainerStarted","Data":"81b266645f022c93bcaa424f1f6a9913bb476b091185d83906084d744c73279c"}
Oct 30 00:35:02.180995 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:02.180961    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7vsmg" event={"ID":"b7f00b4a-044e-49ab-8f43-4351590692e8","Type":"ContainerStarted","Data":"53f1621289f5ad3c9fcaeae350740d13f3f6b7cb39593a65dc2264873ea6bd83"}
Oct 30 00:35:03.194053 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:03.194012    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wb6st" event={"ID":"3a145ece-ad48-43c2-a259-b9590802cfe8","Type":"ContainerStarted","Data":"254fcedd67aba3a971dc33f39c021d8fd0b27aa5b58200266667199db2b73b61"}
Oct 30 00:35:03.194525 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:03.194077    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wb6st" event={"ID":"3a145ece-ad48-43c2-a259-b9590802cfe8","Type":"ContainerStarted","Data":"35275698f3bb8d26780f79ba5b475cdf16da3c96041f071246cafd33235ed56e"}
Oct 30 00:35:03.249692 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:03.249627    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wb6st" podStartSLOduration=66.314435534 podStartE2EDuration="1m7.249608014s" totalImagesPullingTime="935.17248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:03.242072039 +0000 UTC m=+68.114828256" watchObservedRunningTime="2026-10-30 00:35:03.249608014 +0000 UTC m=+68.122364241"
Oct 30 00:35:05.205221 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:05.205178    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-62gpq" event={"ID":"c9803898-12ec-4a07-9a8f-1eef3f89235b","Type":"ContainerStarted","Data":"6b0c46fff81530378bae85fc26329ad83a91a9c4bd73cf4fa6a20ca1625277cc"}
Oct 30 00:35:05.205726 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:05.205400    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:35:05.249073 ip-10-0-130-245 kubenswrapper[2581]: I1030 00:35:05.249022    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-62gpq" podStartSLOduration=66.166877615 podStartE2EDuration="1m9.249006812s" totalImagesPullingTime="3.082129197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:05.248558093 +0000 UTC m=+70.121314317" watchObservedRunningTime="2026-10-30 00:35:05.249006812 +0000 UTC m=+70.121763078"
Oct 30 00:35:06.073998 ip-10-0-130-245 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 00:35:06.091065 ip-10-0-130-245 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 00:35:06.091224 ip-10-0-130-245 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 00:35:06.091383 ip-10-0-130-245 systemd[1]: kubelet.service: Consumed 4.509s CPU time.
Oct 30 00:35:06.110208 ip-10-0-130-245 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 00:35:06.171337 ip-10-0-130-245 kubenswrapper[6099]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:35:06.171337 ip-10-0-130-245 kubenswrapper[6099]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 00:35:06.171337 ip-10-0-130-245 kubenswrapper[6099]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:35:06.171337 ip-10-0-130-245 kubenswrapper[6099]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:35:06.173373 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173297    6099 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:35:06.173373 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173372    6099 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173378    6099 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173383    6099 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173386    6099 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173389    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173391    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173394    6099 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173397    6099 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173400    6099 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173402    6099 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173405    6099 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173408    6099 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173411    6099 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173413    6099 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173416    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173419    6099 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173421    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173424    6099 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173426    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:35:06.173442 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173429    6099 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173433    6099 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173435    6099 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173438    6099 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173440    6099 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173443    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173453    6099 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173456    6099 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173459    6099 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173461    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173464    6099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173466    6099 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173469    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173471    6099 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173474    6099 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173476    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173480    6099 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173483    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173485    6099 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173488    6099 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:35:06.173966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173490    6099 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173493    6099 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173496    6099 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173498    6099 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173501    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173503    6099 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173506    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173509    6099 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173512    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173514    6099 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173517    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173520    6099 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173522    6099 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173526    6099 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173528    6099 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173531    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173533    6099 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173536    6099 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173539    6099 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173542    6099 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:35:06.174459 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173544    6099 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173547    6099 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173549    6099 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173552    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173554    6099 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173557    6099 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173559    6099 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173562    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173564    6099 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173567    6099 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173569    6099 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173572    6099 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173574    6099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173577    6099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173580    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173582    6099 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173585    6099 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173588    6099 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173592    6099 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:35:06.174966 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173595    6099 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173598    6099 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173601    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173603    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173606    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173609    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173612    6099 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173614    6099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173618    6099 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173620    6099 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173623    6099 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173626    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173628    6099 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173631    6099 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173634    6099 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173636    6099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173643    6099 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173646    6099 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173648    6099 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:35:06.175429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173651    6099 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173653    6099 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173656    6099 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173659    6099 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173662    6099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173664    6099 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173666    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.173669    6099 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174419    6099 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174426    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174430    6099 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174434    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174437    6099 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174439    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174442    6099 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174445    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174447    6099 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174450    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174453    6099 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:35:06.175941 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174455    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174458    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174461    6099 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174464    6099 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174466    6099 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174469    6099 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174472    6099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174474    6099 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174480    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174483    6099 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174488    6099 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174491    6099 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174493    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174496    6099 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174499    6099 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174502    6099 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174506    6099 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174508    6099 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174511    6099 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174513    6099 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:35:06.176410 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174516    6099 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174519    6099 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174521    6099 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174524    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174526    6099 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174529    6099 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174532    6099 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174534    6099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174537    6099 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174540    6099 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174542    6099 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174545    6099 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174549    6099 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174552    6099 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174557    6099 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174560    6099 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174563    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174565    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174568    6099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:35:06.176909 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174570    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174573    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174576    6099 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174580    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174582    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174585    6099 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174587    6099 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174590    6099 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174593    6099 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174596    6099 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174598    6099 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174601    6099 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174603    6099 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174606    6099 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174608    6099 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174611    6099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174614    6099 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174616    6099 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174619    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:35:06.177394 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174621    6099 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174624    6099 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174626    6099 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174629    6099 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174632    6099 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174634    6099 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174637    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174639    6099 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174642    6099 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174644    6099 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174647    6099 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174649    6099 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174652    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174655    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174657    6099 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174660    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174664    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174667    6099 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174671    6099 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174674    6099 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:35:06.177900 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174677    6099 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174679    6099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174682    6099 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174685    6099 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174687    6099 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174690    6099 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174692    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174695    6099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174697    6099 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174700    6099 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174720    6099 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174725    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174727    6099 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174730    6099 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174732    6099 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174735    6099 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.174738    6099 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175182    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175188    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175191    6099 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175195    6099 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:35:06.178398 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175197    6099 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175200    6099 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175203    6099 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175207    6099 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175210    6099 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175212    6099 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175215    6099 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175219    6099 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175221    6099 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175224    6099 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175227    6099 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175230    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175232    6099 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175235    6099 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175237    6099 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175240    6099 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175242    6099 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175245    6099 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175248    6099 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175251    6099 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:35:06.178947 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175254    6099 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175257    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175259    6099 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175262    6099 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175264    6099 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175267    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175269    6099 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175272    6099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175274    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175277    6099 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175279    6099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175282    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175284    6099 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175287    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175290    6099 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175294    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175297    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175300    6099 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175303    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:35:06.179466 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175305    6099 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175308    6099 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175311    6099 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175313    6099 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175316    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175318    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175321    6099 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175323    6099 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175326    6099 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175328    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175331    6099 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175334    6099 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175336    6099 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175339    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175342    6099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175344    6099 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175347    6099 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175349    6099 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175352    6099 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175355    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:35:06.179942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175357    6099 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175360    6099 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175362    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175365    6099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175368    6099 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175370    6099 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175373    6099 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175376    6099 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175381    6099 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175385    6099 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175389    6099 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175392    6099 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175395    6099 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175398    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175401    6099 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175404    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175407    6099 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175410    6099 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175413    6099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:35:06.180759 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175416    6099 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175418    6099 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175421    6099 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175424    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175427    6099 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175429    6099 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175432    6099 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175435    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175438    6099 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175440    6099 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175443    6099 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175446    6099 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175448    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175451    6099 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175453    6099 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175456    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175459    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175461    6099 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175465    6099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:35:06.181323 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175467    6099 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175470    6099 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175473    6099 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175475    6099 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.175478    6099 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175563    6099 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175576    6099 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175582    6099 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175586    6099 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175593    6099 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175598    6099 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175605    6099 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175616    6099 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175620    6099 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175623    6099 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175626    6099 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175630    6099 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175635    6099 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175638    6099 flags.go:64] FLAG: --cgroup-root=""
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175641    6099 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175644    6099 flags.go:64] FLAG: --client-ca-file=""
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175647    6099 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175650    6099 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 00:35:06.181791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175654    6099 flags.go:64] FLAG: --cluster-domain=""
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175657    6099 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175660    6099 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175663    6099 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175667    6099 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175671    6099 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175673    6099 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175677    6099 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175680    6099 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175683    6099 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175686    6099 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175689    6099 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175692    6099 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175695    6099 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175700    6099 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175715    6099 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175718    6099 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175721    6099 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175724    6099 flags.go:64] FLAG: --enable-server="true"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175728    6099 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175732    6099 flags.go:64] FLAG: --event-burst="100"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175736    6099 flags.go:64] FLAG: --event-qps="50"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175739    6099 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175742    6099 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 00:35:06.182349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175745    6099 flags.go:64] FLAG: --eviction-hard=""
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175749    6099 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175754    6099 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175757    6099 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175760    6099 flags.go:64] FLAG: --eviction-soft=""
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175763    6099 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175766    6099 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175769    6099 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175772    6099 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175775    6099 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175778    6099 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175781    6099 flags.go:64] FLAG: --feature-gates=""
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175785    6099 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175788    6099 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175791    6099 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175795    6099 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175798    6099 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175801    6099 flags.go:64] FLAG: --help="false"
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175804    6099 flags.go:64] FLAG: --hostname-override="ip-10-0-130-245.ec2.internal"
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175807    6099 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175810    6099 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175813    6099 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175817    6099 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 00:35:06.182949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175820    6099 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175823    6099 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175826    6099 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175829    6099 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175832    6099 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175835    6099 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175839    6099 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175842    6099 flags.go:64] FLAG: --kube-reserved=""
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175845    6099 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175848    6099 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175851    6099 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175855    6099 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175858    6099 flags.go:64] FLAG: --lock-file=""
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175861    6099 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175864    6099 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175868    6099 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175873    6099 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175876    6099 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175879    6099 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175882    6099 flags.go:64] FLAG: --logging-format="text"
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175885    6099 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175889    6099 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175892    6099 flags.go:64] FLAG: --manifest-url=""
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175895    6099 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175900    6099 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 00:35:06.183509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175903    6099 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175907    6099 flags.go:64] FLAG: --max-pods="110"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175910    6099 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175913    6099 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175916    6099 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175920    6099 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175923    6099 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175926    6099 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175929    6099 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175936    6099 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175939    6099 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175942    6099 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175946    6099 flags.go:64] FLAG: --pod-cidr=""
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175950    6099 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175953    6099 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175957    6099 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175960    6099 flags.go:64] FLAG: --port="10250"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175963    6099 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175967    6099 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-031dafe1c982a55ca"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175974    6099 flags.go:64] FLAG: --qos-reserved=""
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175977    6099 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175980    6099 flags.go:64] FLAG: --register-node="true"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175983    6099 flags.go:64] FLAG: --register-with-taints=""
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175987    6099 flags.go:64] FLAG: --registry-burst="10"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175990    6099 flags.go:64] FLAG: --registry-qps="5"
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175993    6099 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 00:35:06.184124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.175996    6099 flags.go:64] FLAG: --reserved-memory=""
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176000    6099 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176003    6099 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176006    6099 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176009    6099 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176012    6099 flags.go:64] FLAG: --runonce="false"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176015    6099 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176018    6099 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176021    6099 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176024    6099 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176027    6099 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176030    6099 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176033    6099 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176036    6099 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176039    6099 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176042    6099 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176045    6099 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176048    6099 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176051    6099 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176054    6099 flags.go:64] FLAG: --system-cgroups=""
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176057    6099 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176062    6099 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176066    6099 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176068    6099 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176074    6099 flags.go:64] FLAG: --tls-min-version=""
Oct 30 00:35:06.184804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176078    6099 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176081    6099 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176084    6099 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176087    6099 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176090    6099 flags.go:64] FLAG: --v="2"
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176095    6099 flags.go:64] FLAG: --version="false"
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176099    6099 flags.go:64] FLAG: --vmodule=""
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176103    6099 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176106    6099 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176260    6099 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176265    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176268    6099 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176270    6099 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176273    6099 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176276    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176278    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176281    6099 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176284    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176286    6099 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176289    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176292    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:35:06.185404 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176294    6099 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176297    6099 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176300    6099 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176302    6099 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176305    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176307    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176310    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176313    6099 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176316    6099 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176319    6099 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176323    6099 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176327    6099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176329    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176332    6099 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176335    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176338    6099 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176340    6099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176343    6099 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176345    6099 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:35:06.185973 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176348    6099 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176351    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176353    6099 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176356    6099 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176358    6099 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176361    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176363    6099 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176366    6099 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176369    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176371    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176374    6099 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176376    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176379    6099 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176381    6099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176384    6099 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176387    6099 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176390    6099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176392    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176394    6099 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176397    6099 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:35:06.186432 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176400    6099 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176404    6099 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176406    6099 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176410    6099 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176414    6099 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176417    6099 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176420    6099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176423    6099 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176426    6099 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176428    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176431    6099 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176434    6099 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176436    6099 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176439    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176442    6099 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176444    6099 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176447    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176451    6099 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176455    6099 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:35:06.187027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176458    6099 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176461    6099 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176464    6099 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176467    6099 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176469    6099 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176472    6099 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176474    6099 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176477    6099 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176481    6099 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176484    6099 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176487    6099 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176490    6099 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176493    6099 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176495    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176498    6099 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176501    6099 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176505    6099 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176508    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176511    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:35:06.187581 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176514    6099 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:35:06.188151 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176516    6099 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:35:06.188151 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176519    6099 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:35:06.188151 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176522    6099 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:35:06.188151 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176525    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:35:06.188151 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176527    6099 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:35:06.188151 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176530    6099 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:35:06.188151 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176532    6099 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:35:06.188151 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176535    6099 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:35:06.188151 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176537    6099 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:35:06.188151 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176540    6099 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:35:06.188151 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176542    6099 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:35:06.188151 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176545    6099 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:35:06.188151 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176548    6099 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:35:06.188151 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176550    6099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:35:06.188151 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176553    6099 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:35:06.188151 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176555    6099 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:35:06.188534 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.176580    6099 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:35:06.188534 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176613    6099 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:35:06.188534 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176618    6099 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:35:06.188534 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176621    6099 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:35:06.188534 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176624    6099 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:35:06.188534 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176627    6099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:35:06.188534 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176629    6099 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:35:06.188534 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176632    6099 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:35:06.188534 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176635    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:35:06.188534 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176637    6099 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:35:06.188534 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176640    6099 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:35:06.188534 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176642    6099 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:35:06.188534 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176645    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:35:06.188534 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176651    6099 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:35:06.188534 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176654    6099 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:35:06.188534 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176657    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176661    6099 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176665    6099 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176669    6099 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176672    6099 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176675    6099 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176678    6099 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176680    6099 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176683    6099 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176685    6099 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176688    6099 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176692    6099 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176695    6099 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176698    6099 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176700    6099 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176714    6099 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176717    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176720    6099 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176722    6099 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:35:06.188960 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176725    6099 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176728    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176730    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176733    6099 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176736    6099 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176739    6099 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176742    6099 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176744    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176747    6099 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176749    6099 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176752    6099 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176758    6099 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176761    6099 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176763    6099 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176766    6099 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176769    6099 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176771    6099 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176774    6099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176777    6099 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176779    6099 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:35:06.189429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176782    6099 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176785    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176787    6099 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176790    6099 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176793    6099 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176795    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176798    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176800    6099 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176803    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176806    6099 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176808    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176811    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176813    6099 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176816    6099 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176819    6099 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176822    6099 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176824    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176827    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176829    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:35:06.190119 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176832    6099 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176835    6099 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176837    6099 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176840    6099 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176846    6099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176850    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176853    6099 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176864    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176868    6099 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176870    6099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176873    6099 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176876    6099 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176878    6099 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176881    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176884    6099 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176886    6099 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176889    6099 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176891    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176894    6099 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176897    6099 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:35:06.190637 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176899    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176902    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176904    6099 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176907    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176909    6099 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176912    6099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176915    6099 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176918    6099 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176921    6099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176924    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176926    6099 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176929    6099 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176931    6099 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.176933    6099 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.181518    6099 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.181533    6099 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.181554    6099 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.181558    6099 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181617    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:35:06.191137 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181623    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181626    6099 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181628    6099 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181631    6099 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181634    6099 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181636    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181639    6099 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181642    6099 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181644    6099 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181647    6099 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181650    6099 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181652    6099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181655    6099 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181657    6099 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181660    6099 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181663    6099 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181665    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181668    6099 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181670    6099 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181673    6099 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:35:06.191594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181675    6099 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181678    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181681    6099 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181684    6099 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181686    6099 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181689    6099 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181691    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181694    6099 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181696    6099 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181699    6099 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181722    6099 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181726    6099 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181728    6099 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181731    6099 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181734    6099 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181736    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181739    6099 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181741    6099 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181744    6099 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181747    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:35:06.192083 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181750    6099 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181752    6099 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181755    6099 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181758    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181760    6099 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181763    6099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181766    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181769    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181772    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181774    6099 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181777    6099 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181779    6099 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181782    6099 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181785    6099 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181787    6099 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181790    6099 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181792    6099 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181795    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181798    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:35:06.192548 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181801    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181804    6099 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181806    6099 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181810    6099 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181814    6099 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181817    6099 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181819    6099 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181822    6099 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181825    6099 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181827    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181830    6099 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181832    6099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181835    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181837    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181840    6099 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181844    6099 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181847    6099 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181850    6099 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181853    6099 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:35:06.193026 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181857    6099 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181860    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181862    6099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181865    6099 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181867    6099 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181870    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181873    6099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181876    6099 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181878    6099 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181881    6099 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181883    6099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181886    6099 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181888    6099 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181890    6099 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181893    6099 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181895    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181898    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181900    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181903    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:35:06.193474 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181905    6099 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:35:06.194027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181908    6099 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:35:06.194027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181911    6099 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:35:06.194027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181914    6099 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:35:06.194027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181916    6099 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:35:06.194027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181919    6099 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:35:06.194027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181921    6099 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:35:06.194027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.181924    6099 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:35:06.194027 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.181952    6099 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:35:06.194027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182081    6099 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:35:06.194027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182085    6099 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:35:06.194027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182088    6099 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:35:06.194027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182091    6099 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:35:06.194027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182095    6099 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:35:06.194027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182100    6099 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:35:06.194027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182103    6099 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:35:06.194027 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182105    6099 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182108    6099 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182110    6099 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182114    6099 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182116    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182119    6099 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182122    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182124    6099 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182127    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182130    6099 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182132    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182135    6099 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182137    6099 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182140    6099 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182143    6099 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182145    6099 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182148    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182150    6099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182153    6099 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:35:06.194419 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182156    6099 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182158    6099 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182161    6099 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182163    6099 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182166    6099 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182169    6099 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182171    6099 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182174    6099 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182177    6099 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182179    6099 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182182    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182185    6099 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182188    6099 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182190    6099 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182193    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182195    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182198    6099 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182200    6099 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182203    6099 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:35:06.194972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182206    6099 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182208    6099 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182211    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182213    6099 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182216    6099 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182218    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182221    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182223    6099 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182226    6099 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182228    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182231    6099 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182233    6099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182236    6099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182238    6099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182241    6099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182243    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182246    6099 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182248    6099 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182251    6099 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:35:06.195431 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182253    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182256    6099 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182258    6099 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182261    6099 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182264    6099 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182266    6099 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182269    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182272    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182274    6099 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182277    6099 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182280    6099 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182282    6099 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182284    6099 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182288    6099 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182291    6099 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182294    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182296    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182299    6099 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182302    6099 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:35:06.195916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182304    6099 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182307    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182309    6099 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182312    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182314    6099 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182317    6099 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182319    6099 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182322    6099 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182324    6099 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182330    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182333    6099 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182335    6099 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182338    6099 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182340    6099 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182343    6099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182346    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182348    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182351    6099 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182354    6099 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182356    6099 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:35:06.196429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182359    6099 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:35:06.197229 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182362    6099 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:35:06.197229 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182365    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:35:06.197229 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.182388    6099 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:35:06.197229 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182420    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:35:06.197229 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182423    6099 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:35:06.197229 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182426    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:35:06.197229 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182430    6099 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:35:06.197229 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182432    6099 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:35:06.197229 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182436    6099 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:35:06.197229 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182439    6099 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:35:06.197229 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182441    6099 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:35:06.197229 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182443    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:35:06.197229 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182446    6099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:35:06.197229 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182449    6099 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:35:06.197229 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182451    6099 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:35:06.197229 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182454    6099 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182456    6099 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182461    6099 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182464    6099 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182466    6099 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182469    6099 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182473    6099 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182476    6099 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182478    6099 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182481    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182484    6099 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182487    6099 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182489    6099 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182492    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182495    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182497    6099 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182500    6099 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182502    6099 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182505    6099 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:35:06.198090 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182508    6099 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182511    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182513    6099 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182516    6099 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182518    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182520    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182523    6099 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182525    6099 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182528    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182530    6099 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182533    6099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182535    6099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182538    6099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182540    6099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182543    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182547    6099 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182550    6099 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182553    6099 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182555    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:35:06.198586 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182559    6099 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182562    6099 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182564    6099 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182567    6099 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182570    6099 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182572    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182575    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182577    6099 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182580    6099 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182583    6099 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182585    6099 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182588    6099 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182591    6099 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182596    6099 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182599    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182601    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182604    6099 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182607    6099 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182609    6099 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:35:06.199111 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182613    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182615    6099 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182618    6099 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182620    6099 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182623    6099 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182625    6099 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182628    6099 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182630    6099 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182633    6099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182637    6099 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182640    6099 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182642    6099 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182645    6099 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182649    6099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182651    6099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182654    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182657    6099 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182659    6099 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182662    6099 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182665    6099 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:35:06.199580 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182668    6099 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182670    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182673    6099 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182675    6099 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182678    6099 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182681    6099 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182685    6099 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182689    6099 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182693    6099 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182696    6099 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182699    6099 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182713    6099 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182718    6099 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182722    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182726    6099 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182729    6099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:06.182732    6099 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.182783    6099 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.183663    6099 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 00:35:06.200096 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.183742    6099 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 00:35:06.200605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.183925    6099 server.go:1014] "Starting client certificate rotation"
Oct 30 00:35:06.200605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.184058    6099 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:35:06.200605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.184112    6099 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:28:56 +0000 UTC" deadline="2028-08-01 10:29:15.77397919 +0000 UTC"
Oct 30 00:35:06.200605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.184145    6099 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15393h54m9.589835615s"
Oct 30 00:35:06.200605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.184380    6099 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:35:06.200605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.186109    6099 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:35:06.200605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.187240    6099 log.go:25] "Validated CRI v1 runtime API"
Oct 30 00:35:06.200605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.189255    6099 log.go:25] "Validated CRI v1 image API"
Oct 30 00:35:06.200605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.189999    6099 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 00:35:06.200605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.192328    6099 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b9afe5ff-e2a3-4f26-9e90-fc8aa7e6a9c6:/dev/nvme0n1p3 f64bb1db-327b-4db0-b50d-925030ff8ef0:/dev/nvme0n1p4]
Oct 30 00:35:06.200900 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.192342    6099 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00ed8e3406e02fbd84b919cd0d099d011999035f9933db5cc62581e6107c63a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00ed8e3406e02fbd84b919cd0d099d011999035f9933db5cc62581e6107c63a1/userdata/shm major:0 minor:317 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/010e66bf2b3c5d50da7b1121c4fc2b867ec2bd71013584ba7dd0c3c6dd07b779/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/010e66bf2b3c5d50da7b1121c4fc2b867ec2bd71013584ba7dd0c3c6dd07b779/userdata/shm major:0 minor:408 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/159cd984f3a29b117fdc8f93c0d301014dfef8c4b4ff50becdc1c6d7b6585fad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/159cd984f3a29b117fdc8f93c0d301014dfef8c4b4ff50becdc1c6d7b6585fad/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2595dae84d252abb34a540cda5b3a7def3276a7c59ef7bd7b8f625c216cc7a1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2595dae84d252abb34a540cda5b3a7def3276a7c59ef7bd7b8f625c216cc7a1c/userdata/shm major:0 minor:277 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2cd2a81a83b1742fb463233f235feabce5febdf65814c37354b4c9f36734834a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2cd2a81a83b1742fb463233f235feabce5febdf65814c37354b4c9f36734834a/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/300b8d79f4ab4d8f825f4849a4f77cee8659de361bab4210aaca9b8fc3e409e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/300b8d79f4ab4d8f825f4849a4f77cee8659de361bab4210aaca9b8fc3e409e8/userdata/shm major:0 minor:281 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fa4df476257e406e2a2ef2135a8a7a2d2f203d6bbd313427afb78fe163973a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fa4df476257e406e2a2ef2135a8a7a2d2f203d6bbd313427afb78fe163973a0/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53f1621289f5ad3c9fcaeae350740d13f3f6b7cb39593a65dc2264873ea6bd83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53f1621289f5ad3c9fcaeae350740d13f3f6b7cb39593a65dc2264873ea6bd83/userdata/shm major:0 minor:534 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/544c576d92b15de7c0ede662e9e7824f6378d9bbe565085dad652bcb21bfdce7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/544c576d92b15de7c0ede662e9e7824f6378d9bbe565085dad652bcb21bfdce7/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f3c807b145f40d120e4ae924c62641da716306c354344fb78151aabbea7d01e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f3c807b145f40d120e4ae924c62641da716306c354344fb78151aabbea7d01e/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7bab66348e408303b7eb14327330fe90dc78ec9222ea1bf64848cc0d9b162120/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7bab66348e408303b7eb14327330fe90dc78ec9222ea1bf64848cc0d9b162120/userdata/shm major:0 minor:375 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81b266645f022c93bcaa424f1f6a9913bb476b091185d83906084d744c73279c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81b266645f022c93bcaa424f1f6a9913bb476b091185d83906084d744c73279c/userdata/shm major:0 minor:536 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82479f721db5be7196aa5f19098e38b102f9fb744d55881c21733f3bdd59f635/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82479f721db5be7196aa5f19098e38b102f9fb744d55881c21733f3bdd59f635/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8691cb9acb1869b233346dbd05a670fea0cf9c382094432b55288b91d83852c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8691cb9acb1869b233346dbd05a670fea0cf9c382094432b55288b91d83852c2/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9070bded93cff1254d1950c7952ed0da996728ddbbbf7b7aef1623fbe8630bc8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9070bded93cff1254d1950c7952ed0da996728ddbbbf7b7aef1623fbe8630bc8/userdata/shm major:0 minor:532 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2f3cc0b7891f938dcbe4b74b6e5d04be4e232fd453c9d65e55304d591f42f56/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2f3cc0b7891f938dcbe4b74b6e5d04be4e232fd453c9d65e55304d591f42f56/userdata/shm major:0 minor:279 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a58bdd63ee44dcdfe20b21c5471578b608e708d7aaa3c13f4c20b5f89c390711/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a58bdd63ee44dcdfe20b21c5471578b608e708d7aaa3c13f4c20b5f89c390711/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab0c9e3bbd550a22c57907f05402ac9314905c0c759dec1a19a3abf77b32903e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab0c9e3bbd550a22c57907f05402ac9314905c0c759dec1a19a3abf77b32903e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad53fa1332f37c8b6761caff50c30eba5a7ae08ad8928e0fcbdc157d85629928/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad53fa1332f37c8b6761caff50c30eba5a7ae08ad8928e0fcbdc157d85629928/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/adcc8ffdae79afac97367613656572dba910587bee16e642ad90bc6ff4f6fa93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/adcc8ffdae79afac97367613656572dba910587bee16e642ad90bc6ff4f6fa93/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bbdeeec904dc0720fe0f45061c51cf73759829349eb4521341d423a42cf61960/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bbdeeec904dc0720fe0f45061c51cf73759829349eb4521341d423a42cf61960/userdata/shm major:0 minor:430 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc17043616d9ef84669e5f37b1297765b0dfc2cc919ce4683ea5831135982716/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc17043616d9ef84669e5f37b1297765b0dfc2cc919ce4683ea5831135982716/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ccbb805d5abfb4915455ccbb72ea088e0952c14325b69d672c20b45e6fcec5c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ccbb805d5abfb4915455ccbb72ea088e0952c14325b69d672c20b45e6fcec5c8/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9ad9b60e9424a59c50647f75bc4c95b224255b4e983bf3a664f5b06e50e507c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9ad9b60e9424a59c50647f75bc4c95b224255b4e983bf3a664f5b06e50e507c/userdata/shm major:0 minor:472 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0abdd59e265409683f86e9f7c116e9998805263b62af79330f4dd475ca46a13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0abdd59e265409683f86e9f7c116e9998805263b62af79330f4dd475ca46a13/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/edaebab344eb8b504f4cbb62b8b245ce7e6a502dfbe2011df04d7030ddd7e6e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/edaebab344eb8b504f4cbb62b8b245ce7e6a502dfbe2011df04d7030ddd7e6e8/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1af4bcd067d00480932bbef7d3d95d8577e178a33a277d7ab5d7b1b8c1c0ddd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1af4bcd067d00480932bbef7d3d95d8577e178a33a277d7ab5d7b1b8c1c0ddd/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volume-subpaths/web-config/config-reloader/2 major:0 minor:464 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:464 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volume-subpaths/web-config/prometheus/15 major:0 minor:464 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~empty-dir/config-out major:0 minor:466 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~projected/kube-api-access-j5kb7:{mountpoint:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~projected/kube-api-access-j5kb7 major:0 minor:471 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~projected/tls-assets major:0 minor:468 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/config major:0 minor:467 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:461 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:456 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:462 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:470 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:463 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:469 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:465 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/web-config major:0 minor:464 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/070879ac-09e8-4fdb-b55f-e52a9998552f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/070879ac-09e8-4fdb-b55f-e52a9998552f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/070879ac-09e8-4fdb-b55f-e52a9998552f/volumes/kubernetes.io~projected/kube-api-access-257nk:{mountpoint:/var/lib/kubelet/pods/070879ac-09e8-4fdb-b55f-e52a9998552f/volumes/kubernetes.io~projected/kube-api-access-257nk major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/070879ac-09e8-4fdb-b55f-e52a9998552f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/070879ac-09e8-4fdb-b55f-e52a9998552f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b308ab3-9eb7-4be8-964f-ff94af40b883/volumes/kubernetes.io~projected/kube-api-access-6j86q:{mountpoint:/var/lib/kubelet/pods/0b308ab3-9eb7-4be8-964f-ff94af40b883/volumes/kubernetes.io~projected/kube-api-access-6j86q major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b308ab3-9eb7-4be8-964f-ff94af40b883/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0b308ab3-9eb7-4be8-964f-ff94af40b883/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12bf7b6a-fd62-4921-910d-f9fff2b347d1/volumes/kubernetes.io~projected/kube-api-access-j7spz:{mountpoint:/var/lib/kubelet/pods/12bf7b6a-fd62-4921-910d-f9fff2b347d1/volumes/kubernetes.io~projected/kube-api-access-j7spz major:0 minor:399 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12bf7b6a-fd62-4921-910d-f9fff2b347d1/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/12bf7b6a-fd62-4921-910d-f9fff2b347d1/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:393 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21be6874-6138-4a00-9c6e-aa1841eb3230/volumes/kubernetes.io~projected/kube-api-access-p7lkm:{mountpoint:/var/lib/kubelet/pods/21be6874-6138-4a00-9c6e-aa1841eb3230/volumes/kubernetes.io~projected/kube-api-access-p7lkm major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21be6874-6138-4a00-9c6e-aa1841eb3230/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/21be6874-6138-4a00-9c6e-aa1841eb3230/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21be6874-6138-4a00-9c6e-aa1841eb3230/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/21be6874-6138-4a00-9c6e-aa1841eb3230/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6/volumes/kubernetes.io~projected/kube-api-access-xzmdn:{mountpoint:/var/lib/kubelet/pods/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6/volumes/kubernetes.io~projected/kube-api-access-xzmdn major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32086945-d23d-49bb-a876-fab4cef52ec6/volumes/kubernetes.io~projected/kube-api-access-d9rmw:{mountpoint:/var/lib/kubelet/pods/32086945-d23d-49bb-a876-fab4cef52ec6/volumes/kubernetes.io~projected/kube-api-access-d9rmw major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a145ece-ad48-43c2-a259-b9590802cfe8/volumes/kubernetes.io~projected/kube-api-access-8nbn8:{mountpoint:/var/lib/kubelet/pods/3a145ece-ad48-43c2-a259-b9590802cfe8/volumes/kubernetes.io~projected/kube-api-access-8nbn8 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a145ece-ad48-43c2-a259-b9590802cfe8/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/3a145ece-ad48-43c2-a259-b9590802cfe8/volumes/kubernetes.io~secret/metrics-certs major:0 minor:530 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volume-subpaths/web-config/alertmanager/9 major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volume-subpaths/web-config/config-reloader/6 major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~empty-dir/config-out major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~projected/kube-api-access-fwm8g:{mountpoint:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~projected/kube-api-access-fwm8g major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~projected/tls-assets major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/config-volume major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/web-config major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47d34735-e4b1-4073-b6de-98a9a720432b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/47d34735-e4b1-4073-b6de-98a9a720432b/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5975a832-beb3-4e76-98ad-aef36a2ccce2/volumes/kubernetes.io~projected/kube-api-access-cnd4w:{mountpoint:/var/lib/kubelet/pods/5975a832-beb3-4e76-98ad-aef36a2ccce2/volumes/kubernetes.io~projected/kube-api-access-cnd4w major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ac72880-98ab-4093-b7aa-903a37c8113e/volumes/kubernetes.io~projected/kube-api-access-bmqr7:{mountpoint:/var/lib/kubelet/pods/5ac72880-98ab-4093-b7aa-903a37c8113e/volumes/kubernetes.io~projected/kube-api-access-bmqr7 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3/volumes/kubernetes.io~projected/kube-api-access-fpbnj:{mountpoint:/var/lib/kubelet/pods/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3/volumes/kubernetes.io~projected/kube-api-access-fpbnj major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/733527bc-83b7-423c-a79e-1de279f5a607/volumes/kubernetes.io~projected/kube-api-access-jhvbj:{mountpoint:/var/lib/kubelet/pods/733527bc-83b7-423c-a79e-1de279f5a607/volumes/kubernetes.io~projected/kube-api-access-jhvbj major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/962a4e53-88b6-4866-abc5-108b6d5ce784/volumes/kubernetes.io~projected/kube-api-access-lhjkw:{mountpoint:/var/lib/kubelet/pods/962a4e53-88b6-4866-abc5-108b6d5ce784/volumes/kubernetes.io~projected/kube-api-access-lhjkw major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/962a4e53-88b6-4866-abc5-108b6d5ce784/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/962a4e53-88b6-4866-abc5-108b6d5ce784/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/962a4e53-88b6-4866-abc5-108b6d5ce784/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/962a4e53-88b6-4866-abc5-108b6d5ce784/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7/volumes/kubernetes.io~projected/kube-api-access-f4tjg:{mountpoint:/var/lib/kubelet/pods/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7/volumes/kubernetes.io~projected/kube-api-access-f4tjg major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b73f4c46-cb05-492e-8dbc-953f3c70b1d5/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b73f4c46-cb05-492e-8dbc-953f3c70b1d5/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b73f4c46-cb05-492e-8dbc-953f3c70b1d5/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b73f4c46-cb05-492e-8dbc-953f3c70b1d5/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b73f4c46-cb05-492e-8dbc-953f3c70b1d5/volumes/kubernetes.io~projected/kube-api-access-f24vm:{mountpoint:/var/lib/kubelet/pods/b73f4c46-cb05-492e-8dbc-953f3c70b1d5/volumes/kubernetes.io~projected/kube-api-access-f24vm major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7f00b4a-044e-49ab-8f43-4351590692e8/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b7f00b4a-044e-49ab-8f43-4351590692e8/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:526 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8c48d07-5599-4a74-94d4-d6488bde2827/volumes/kubernetes.io~projected/kube-api-access-qmnmx:{mountpoint:/var/lib/kubelet/pods/b8c48d07-5599-4a74-94d4-d6488bde2827/volumes/kubernetes.io~projected/kube-api-access-qmnmx major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8c48d07-5599-4a74-94d4-d6488bde2827/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/b8c48d07-5599-4a74-94d4-d6488bde2827/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8c48d07-5599-4a74-94d4-d6488bde2827/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/b8c48d07-5599-4a74-94d4-d6488bde2827/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8c48d07-5599-4a74-94d4-d6488bde2827/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/b8c48d07-5599-4a74-94d4-d6488bde2827/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc8e4d97-4c7f-450d-961c-528c992a29b2/volumes/kubernetes.io~projected/kube-api-access-mkp59:{mountpoint:/var/lib/kubelet/pods/bc8e4d97-4c7f-450d-961c-528c992a29b2/volumes/kubernetes.io~projected/kube-api-access-mkp59 major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc8e4d97-4c7f-450d-961c-528c992a29b2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/bc8e4d97-4c7f-450d-961c-528c992a29b2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc8e4d97-4c7f-450d-961c-528c992a29b2/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/bc8e4d97-4c7f-450d-961c-528c992a29b2/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~projected/kube-api-access-wpw5w:{mountpoint:/var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~projected/kube-api-access-wpw5w major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9803898-12ec-4a07-9a8f-1eef3f89235b/volumes/kubernetes.io~projected/kube-api-access-qbhmk:{mountpoint:/var/lib/kubelet/pods/c9803898-12ec-4a07-9a8f-1eef3f89235b/volumes/kubernetes.io~projected/kube-api-access-qbhmk major:0 minor:531 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d996f136-ba95-4636-a870-94520d3eff0f/volumes/kubernetes.io~projected/kube-api-access-vbtjm:{mountpoint:/var/lib/kubelet/pods/d996f136-ba95-4636-a870-94520d3eff0f/volumes/kubernetes.io~projected/kube-api-access-vbtjm major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28/volumes/kubernetes.io~projected/kube-api-access-9ncsp:{mountpoint:/var/lib/kubelet/pods/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28/volumes/kubernetes.io~projected/kube-api-access-9ncsp major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:426 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f53ec687-40dd-4358-bc05-83d064d9d082/volumes/kubernetes.io~projected/kube-api-access-l6gmn:{mountpoint:/var/lib/kubelet/pods/f53ec687-40dd-4358-bc05-83d064d9d082/volumes/kubernetes.io~projected/kube-api-access-l6gmn major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f53ec687-40dd-4358-bc05-83d064d9d082/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f53ec687-40dd-4358-bc05-83d064d9d082/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f53ec687-40dd-4358-bc05-83d064d9d082/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/f53ec687-40dd-4358-bc05-83d064d9d082/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd/volumes/kubernetes.io~projected/kube-api-access-r59hp:{mountpoint:/var/lib/kubelet/pods/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd/volumes/kubernetes.io~projected/kube-api-access-r59hp major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:285 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a4c133b04e33897fcb89da89320e4bcd0f2874bb82497bd1b510aac94dad7d5c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/56ec608bcca49ff27345b38575c856aa283672afb756e42258ee5030f5a797bd/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/be8d9b04a8974e0d4b5673f38de5ccff9da02e4bf07588f4731d4d789b1bce6d/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/a52787106932ae4183f0007456f840eadb9e9a16ff60d0a10f9d23c339cfdeb6/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/b44c6e9f1971d2d27f788b65168622a13f8af5674085754a37a00e49231ab6e3/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/9cd64a407cb18c8d2a42ad123550caed1d5177f57d29e98ab251d9b90e1d8107/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/d355e9a548d57ce8d2654959edda2f6cc7b6dba685d14ba1a4e11ac75d890bdc/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f70a28fb3e5c097d2a5e876bf1ed15d91a89eff5bcb21a3284cd4f39f84d9410/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/be71cca2e378b25dfa0b74053f924eda5f7e90e5fca3fa639d7343837f3ccd28/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/587febcd452edc44fef5182b19ee13f30b4edea4f1cfb2432208611a45af361b/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/aaec66717317187ee62683e6a9ce330c64ab6d9400aede37e91534db73613b93/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/cb834b547a43a96695cffd856c4563624a21c85f4fa41af2d6a248c3ff847c01/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/1adeef74a301fa3e13815dfbc62938e14efa355d51819bd66399c8d320c6e208/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/f30360c82004e68e7319e8133e8a877b2f6075f7a4a7012aea1c179fae6601a4/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/114011d322e97ade66725f9743792959a08e0e6ea8a392935fc211a8c65614f8/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/5150aa92af83dc67c6689ce3474711aae6f5f3d99fa42d96a674718b87b749e4/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8aa0082c9c35ff1f0e8cba5b7e46b0c421bc22a056e52427a11fce7eef943595/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/cf79a64c00e9a621f94d63240e0772ec5f2532cd50a2c2ef24074d1cec932075/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/bb9fe39e46ddd017f7bd3735c60dcb3042cc72c57688836c8152500ffdb224f1/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/467702b534d0bd26849a5eee281fbcaa97b0b1b8dd4bfc9750b97d7cbe9a9061/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/206d4876b34e145590971c5fff5e82f40074f1bc2b7090b8bed981c303ae9108/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/9d59ee88af0282d7c0f48c936688b7c48c0be11a5f902cd09ab4d0aea6ff99b7/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/4170ee66176be02848a541efe2cabd53d75b1813f87c1417545f33c3ec43deae/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/1473acba814de865c53c384a5610c17d702290f48f49d11a9e0050a2fbae4a85/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/7621ef2eb1390ad502f9e0536f6b2e3a3864f5d8b039c0432e594018a7e64dde/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/1d0c5a089a9fd1998a79f2b30b0b42dfa19df4ac64bf3f0311ab12be5c543348/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/f8623f8121252a85003a53496ac5fb75429226cfedcdc25f8fac706878a85ba9/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/624fa31800ca3c30fb025b471265ace45993ea3329f4805e58aceb4eda878620/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/67d539f65c123a433ecd505f9d12dd0a3e2064ea684e0c7b0869caa6d288851a/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/373f70da082d177d6ee8c692950603a48fc92c6ef72f13d2582ed68eab09a59d/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/5b8a8554de933066673aa980ec35fb734844bce900a2bd3485e3f3042dffd07d/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/ff8af21863f42079773c70b325749ec01487ca21da02e5f5868b52f20cfd7d7f/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/f11fd79551622ca97577d6894cc8e13d66f84738763c4d3da79805c46ee56336/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/eb184629142e44f85904229e3ae0b5a9619477ce42d25d93cf1f534e0d1482e1/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/097886e4f561be6c5d7d9409cd9d2c0d63d8129fa76617c6d12da1c2507e9ab3/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/43bc984fdf7cf832185159279b2f0175f4ae0d5a1419e37aa790dad58cac427c/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/30dec416911eaaf84517c5f2995df5d795ba40012f6575aa9ce773bc079b519b/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/8dd5804c302d700196cf9b06cb21a4f3d8c6b174d7a2049276a121b37cf922a6/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/58cb1e1dbdbf75cab27c436bacd7588fbdca3735a9f3a8a15d4d3150b32481d8/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/585847cbe0b055929f25633720d4b4f4074239b1e9e81d8f6ad766016cde1062/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/85076384d6e6ed7111b0c9e4ddcb20ca73f0a92cec33c15fe46f47e078a43103/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/ad55fca36627b5dc13e013935a0319e0bd96b7ef8fc27c39c39fd1c0c8944614/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/199ad9b921d0452377e7fdf3560f7f2f369b16ecb90faadf80d08df9d319550c/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/162a412a4019aa1a8d2b9d392f3c4013365e2def963dfc600379edef601e4627/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/93eb3bf2fabb148019843d15d0bfcaa34b8746625fff3351154dd94ce4607959/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/7890147668d863d5a15458a67e526f3355f128872e9656b5c31d5a59518abf3b/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/971dce70e0f03fba96af803bdcf3da468c06cb4da8bc78e1013f2a09bc30feae/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/ebd48993e4b50cc3b4c77bbbfd916631bf7610e3ecc3a4e8f00634a7912e574f/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/a28f9e14d85730d3714e01eee7c140f4e8cf290de9e19ccc36364f5ce1d6e5e2/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/b22ed7fc629633cd55f40cdfe87d39155c3cbf541f09538d606fb4cba7de5851/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/3bf58db69c1aec5a87fdaf4fd84beeac5949b39f3753a3a6d9943900ef3cf808/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/e1fd18b9613a660c394d3e2340f67b4da49060697c088c08786d73ded64077be/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/22d0687ac7317e0ee7e19e85838ef00d2e6b6c0a8f4e911dbf21d55816ab05da/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/ce1498851d929fdf04564e55e8a06af67a011c1536af0aa148e982b2058fb243/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/b1ee89ee74f4cf14828b593c018c0d5fcb2e5576d2645174b5224b45ee80fb56/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/fd69f4f5fe2b34843c212dafe1319e8173348755157c0f4fc45c2b5849d69973/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/82af3a6080ab21e55b43b551aa58bcacd7c46ecc27ae0cd5533921f4c19fb608/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/ea5aa6395f951ddacf49e564e649433720b5b178f9a39f5da1400a4c502a7e7b/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/8d4da2d666d969d8d2540a25dd129c2bfdc5b030e56fc32accced11f0789923d/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/576576ea7ebab6c276ae21c96fc2b910122f02873089d3ffb7ed9c984489be68/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/08605fe484795f22832fb10506a9717b0784d183ef9353418aa762129fa0bb52/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/9b482af87b7c4f61a0c125a67c8354d9e2e6042ad03953b6572f5f262f657141/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-446:{mountpoint:/var/lib/containers/storage/overlay/f6c89d2f0149ce55efae47d520b2e9a9396f1efc785517d3289c7fd48ea826b3/merged major:0 minor:446 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/e2fab57daa9645d2dadd64ac7f40fd264f3013a79e6d2d9a46f5cefac6a20c83/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-474:{mountpoint:/var/lib/containers/storage/overlay/955ea2500bacfe16564a376d2a9e96d206c2fef7400bd66e0af01f1c9659c953/merged major:0 minor:474 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/c3e7161aba03bfe9efa412c3e9a60053cb0e218c6aace5b38662b48340c6d3c8/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/71e2377d1606c4824c59646d8454cd91b74207526d0a4c5b01ea8c2d3eeebb40/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-484:{mountpoint:/var/lib/containers/storage/overlay/84fdedce0098779f9635fd45223e157f3e6227d90d345b99ea04dff0b156f5ca/merged major:0 minor:484 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/532a2b0255c91f40c985d7e5d0b151d6756e78783c81f99df40665fe65bec024/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-490:{mountpoint:/var/lib/containers/storage/overlay/6ba49d8a59572798c47a07d9c9a7568377cc020813417ba66b76004cf8b1c7e7/merged major:0 minor:490 fsType:overlay blockSize:0} overlay_0-495:{mountpoint:/var/lib/containers/storage/overlay/a5d55c584fe1ca7b200c148d15c011198757ca6274ca27af4b6ddeb620311448/merged major:0 minor:495 fsType:overlay blockSize:0} overlay_0-500:{mountpoint:/var/lib/containers/storage/overlay/bac22e43377bd5dd015d7dd1a8ffcaf783feefb47af5c68399960a6526e6d3ab/merged major:0 minor:500 fsType:overlay blockSize:0} overlay_0-506:{mountpoint:/var/lib/containers/storage/overlay/cffe274d9bcd3c8a9691f4a0624d99c0a8a5da82b7e2e57e1325f3d2e04a90a9/merged major:0 minor:506 fsType:overlay blockSize:0} overlay_0-511:{mountpoint:/var/lib/containers/storage/overlay/4fcb0550b5f0f9aeac7ed72be6be3ac6ec5ab4e6bd41833d93f0e3f266aef636/merged major:0 minor:511 fsType:overlay blockSize:0} overlay_0-516:{mountpoint:/var/lib/containers/storage/overlay/03f3501c41559a14c111a9679cda92d2772e179d7426b331ae8564dc275b33e8/merged major:0 minor:516 fsType:overlay blockSize:0} overlay_0-521:{mountpoint:/var/lib/containers/storage/overlay/cb3eabe8136c0e27794f9a4b3f746bfd91477f0fdddf7457565663293daad564/merged major:0 minor:521 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/11efa55edf834d5278866f71550ef190ced2593c701e536bd0edfde8a41cd46d/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-538:{mountpoint:/var/lib/containers/storage/overlay/a91c97da5f674d97736e7ea78b591384c85d8ce3fe170a91567ec63ef2a72409/merged major:0 minor:538 fsType:overlay blockSize:0} overlay_0-540:{mountpoint:/var/lib/containers/storage/overlay/d69e14a4c3e448143cfa80d36a3080ab60e647bfae9c30224cee95719f9c2285/merged major:0 minor:540 fsType:overlay blockSize:0} overlay_0-542:{mountpoint:/var/lib/containers/storage/overlay/e444ddf0dda4fae949b2d6d6de3b5dcd65e08392ae8fd45a080ef6a78b68f321/merged major:0 minor:542 fsType:overlay blockSize:0} overlay_0-544:{mountpoint:/var/lib/containers/storage/overlay/5d34a21e27ada452d3fed5445876600f93a4a9b9a86808cd2b9278fbacc88f3c/merged major:0 minor:544 fsType:overlay blockSize:0} overlay_0-546:{mountpoint:/var/lib/containers/storage/overlay/ad3e10dcb721d127fa6fd69728cb583b78c85d305e1608f111cda4c95ce5128a/merged major:0 minor:546 fsType:overlay blockSize:0} overlay_0-552:{mountpoint:/var/lib/containers/storage/overlay/4bdcb12b4a11a0341dc3658e9ea7d750c9db4a473b65220e635cb5765dc1a0da/merged major:0 minor:552 fsType:overlay blockSize:0} overlay_0-557:{mountpoint:/var/lib/containers/storage/overlay/d4e5b6f0c134b3fcc286ada706fed4cffd274886a8642e0c3d7b5907b3c57d21/merged major:0 minor:557 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/8e9532af094d0bcf52b6580aa06327f00fafa0958563e865e1a87428dbca2637/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/51d84250987454a38b0ecb56ac4189f981dfcaac78a7c5903e892913e143d424/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/abba117bc1a958e57db50e3abc1e085c7b1255fa7f8a9ddfc32e32c09991f9ca/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/652ad4632911be253be4fe2e5726e098c634f8de7a49bd5121e9164de15cb51b/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/e040f5d27064d85635e4adf8cbe290049507020abf55fe2a52b08b3f638c4ff8/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/19b664ec3c5b8f2562fe1ba3595712138d070308e4ed6bf0491fdd72630c1fd2/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/0d16ad769bc78d9a40e64ca6c7c3b32008192c97f5379022654c3c01bb189349/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/a10f6831e416f7587364e8eeb1c22f70035a56b5faf9568b7014d2368dc9462c/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 00:35:06.202334 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.201828    6099 manager.go:217] Machine: {Timestamp:2026-10-30 00:35:06.201341069 +0000 UTC m=+0.058248818 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093034 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ecc5a2129e52e70e688440f2119ed SystemUUID:ec2ecc5a-2129-e52e-70e6-88440f2119ed BootID:e1d50569-5e03-478a-80bc-e3f387eb528e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21be6874-6138-4a00-9c6e-aa1841eb3230/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:241 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bc8e4d97-4c7f-450d-961c-528c992a29b2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:270 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:313 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-516 DeviceMajor:0 DeviceMinor:516 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-544 DeviceMajor:0 DeviceMinor:544 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc8e4d97-4c7f-450d-961c-528c992a29b2/volumes/kubernetes.io~projected/kube-api-access-mkp59 DeviceMajor:0 DeviceMinor:275 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:307 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:313 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/733527bc-83b7-423c-a79e-1de279f5a607/volumes/kubernetes.io~projected/kube-api-access-jhvbj DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b73f4c46-cb05-492e-8dbc-953f3c70b1d5/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0b308ab3-9eb7-4be8-964f-ff94af40b883/volumes/kubernetes.io~projected/kube-api-access-6j86q DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-474 DeviceMajor:0 DeviceMinor:474 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ac72880-98ab-4093-b7aa-903a37c8113e/volumes/kubernetes.io~projected/kube-api-access-bmqr7 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3/volumes/kubernetes.io~projected/kube-api-access-fpbnj DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:312 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:464 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7/volumes/kubernetes.io~projected/kube-api-access-f4tjg DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d996f136-ba95-4636-a870-94520d3eff0f/volumes/kubernetes.io~projected/kube-api-access-vbtjm DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/070879ac-09e8-4fdb-b55f-e52a9998552f/volumes/kubernetes.io~projected/kube-api-access-257nk DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b8c48d07-5599-4a74-94d4-d6488bde2827/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:365 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-500 DeviceMajor:0 DeviceMinor:500 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a145ece-ad48-43c2-a259-b9590802cfe8/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:530 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a58bdd63ee44dcdfe20b21c5471578b608e708d7aaa3c13f4c20b5f89c390711/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:315 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bbdeeec904dc0720fe0f45061c51cf73759829349eb4521341d423a42cf61960/userdata/shm DeviceMajor:0 DeviceMinor:430 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6/volumes/kubernetes.io~projected/kube-api-access-xzmdn DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f53ec687-40dd-4358-bc05-83d064d9d082/volumes/kubernetes.io~projected/kube-api-access-l6gmn DeviceMajor:0 DeviceMinor:274 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:353 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9ad9b60e9424a59c50647f75bc4c95b224255b4e983bf3a664f5b06e50e507c/userdata/shm DeviceMajor:0 DeviceMinor:472 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-552 DeviceMajor:0 DeviceMinor:552 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a145ece-ad48-43c2-a259-b9590802cfe8/volumes/kubernetes.io~projected/kube-api-access-8nbn8 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/adcc8ffdae79afac97367613656572dba910587bee16e642ad90bc6ff4f6fa93/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~projected/kube-api-access-wpw5w DeviceMajor:0 DeviceMinor:358 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:469 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f53ec687-40dd-4358-bc05-83d064d9d082/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:272 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:357 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2cd2a81a83b1742fb463233f235feabce5febdf65814c37354b4c9f36734834a/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad53fa1332f37c8b6761caff50c30eba5a7ae08ad8928e0fcbdc157d85629928/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:461 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c9803898-12ec-4a07-9a8f-1eef3f89235b/volumes/kubernetes.io~projected/kube-api-access-qbhmk DeviceMajor:0 DeviceMinor:531 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21be6874-6138-4a00-9c6e-aa1841eb3230/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:245 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/12bf7b6a-fd62-4921-910d-f9fff2b347d1/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:393 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/010e66bf2b3c5d50da7b1121c4fc2b867ec2bd71013584ba7dd0c3c6dd07b779/userdata/shm DeviceMajor:0 DeviceMinor:408 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:464 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-511 DeviceMajor:0 DeviceMinor:511 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81b266645f022c93bcaa424f1f6a9913bb476b091185d83906084d744c73279c/userdata/shm DeviceMajor:0 DeviceMinor:536 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/962a4e53-88b6-4866-abc5-108b6d5ce784/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:265 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:285 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7bab66348e408303b7eb14327330fe90dc78ec9222ea1bf64848cc0d9b162120/userdata/shm DeviceMajor:0 DeviceMinor:375 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/962a4e53-88b6-4866-abc5-108b6d5ce784/volumes/kubernetes.io~projected/kube-api-access-lhjkw DeviceMajor:0 DeviceMinor:276 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:349 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:355 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc17043616d9ef84669e5f37b1297765b0dfc2cc919ce4683ea5831135982716/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:463 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:467 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-490 DeviceMajor:0 DeviceMinor:490 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:465 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:464 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-521 DeviceMajor:0 DeviceMinor:521 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8691cb9acb1869b233346dbd05a670fea0cf9c382094432b55288b91d83852c2/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ccbb805d5abfb4915455ccbb72ea088e0952c14325b69d672c20b45e6fcec5c8/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28/volumes/kubernetes.io~projected/kube-api-access-9ncsp DeviceMajor:0 DeviceMinor:427 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:470 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-540 DeviceMajor:0 DeviceMinor:540 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5975a832-beb3-4e76-98ad-aef36a2ccce2/volumes/kubernetes.io~projected/kube-api-access-cnd4w DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:309 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:310 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/edaebab344eb8b504f4cbb62b8b245ce7e6a502dfbe2011df04d7030ddd7e6e8/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1af4bcd067d00480932bbef7d3d95d8577e178a33a277d7ab5d7b1b8c1c0ddd/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0abdd59e265409683f86e9f7c116e9998805263b62af79330f4dd475ca46a13/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:356 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab0c9e3bbd550a22c57907f05402ac9314905c0c759dec1a19a3abf77b32903e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/070879ac-09e8-4fdb-b55f-e52a9998552f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/544c576d92b15de7c0ede662e9e7824f6378d9bbe565085dad652bcb21bfdce7/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b8c48d07-5599-4a74-94d4-d6488bde2827/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:371 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:426 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f3c807b145f40d120e4ae924c62641da716306c354344fb78151aabbea7d01e/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21be6874-6138-4a00-9c6e-aa1841eb3230/volumes/kubernetes.io~projected/kube-api-access-p7lkm DeviceMajor:0 DeviceMinor:246 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:311 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5282344-9beb-485c-9525-6b09bdda0804/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:354 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b8c48d07-5599-4a74-94d4-d6488bde2827/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:370 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b308ab3-9eb7-4be8-964f-ff94af40b883/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2f3cc0b7891f938dcbe4b74b6e5d04be4e232fd453c9d65e55304d591f42f56/userdata/shm DeviceMajor:0 DeviceMinor:279 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/12bf7b6a-fd62-4921-910d-f9fff2b347d1/volumes/kubernetes.io~projected/kube-api-access-j7spz DeviceMajor:0 DeviceMinor:399 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:420 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:313 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:312 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:462 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53f1621289f5ad3c9fcaeae350740d13f3f6b7cb39593a65dc2264873ea6bd83/userdata/shm DeviceMajor:0 DeviceMinor:534 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-542 DeviceMajor:0 DeviceMinor:542 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:425 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:468 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b7f00b4a-044e-49ab-8f43-4351590692e8/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:526 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-557 DeviceMajor:0 DeviceMinor:557 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd/volumes/kubernetes.io~projected/kube-api-access-r59hp DeviceMajor:0 DeviceMinor:287 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:424 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-446 DeviceMajor:0 DeviceMinor:446 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:456 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~projected/kube-api-access-j5kb7 DeviceMajor:0 DeviceMinor:471 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f53ec687-40dd-4358-bc05-83d064d9d082/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:308 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9070bded93cff1254d1950c7952ed0da996728ddbbbf7b7aef1623fbe8630bc8/userdata/shm DeviceMajor:0 DeviceMinor:532 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/47d34735-e4b1-4073-b6de-98a9a720432b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/159cd984f3a29b117fdc8f93c0d301014dfef8c4b4ff50becdc1c6d7b6585fad/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82479f721db5be7196aa5f19098e38b102f9fb744d55881c21733f3bdd59f635/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2595dae84d252abb34a540cda5b3a7def3276a7c59ef7bd7b8f625c216cc7a1c/userdata/shm DeviceMajor:0 DeviceMinor:277 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:314 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00ed8e3406e02fbd84b919cd0d099d011999035f9933db5cc62581e6107c63a1/userdata/shm DeviceMajor:0 DeviceMinor:317 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes/kubernetes.io~projected/kube-api-access-fwm8g DeviceMajor:0 DeviceMinor:316 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:313 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:466 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-484 DeviceMajor:0 DeviceMinor:484 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-506 DeviceMajor:0 DeviceMinor:506 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-546 DeviceMajor:0 DeviceMinor:546 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b73f4c46-cb05-492e-8dbc-953f3c70b1d5/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b73f4c46-cb05-492e-8dbc-953f3c70b1d5/volumes/kubernetes.io~projected/kube-api-access-f24vm DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/32086945-d23d-49bb-a876-fab4cef52ec6/volumes/kubernetes.io~projected/kube-api-access-d9rmw DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/962a4e53-88b6-4866-abc5-108b6d5ce784/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:271 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bc8e4d97-4c7f-450d-961c-528c992a29b2/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:273 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/300b8d79f4ab4d8f825f4849a4f77cee8659de361bab4210aaca9b8fc3e409e8/userdata/shm DeviceMajor:0 DeviceMinor:281 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/070879ac-09e8-4fdb-b55f-e52a9998552f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fa4df476257e406e2a2ef2135a8a7a2d2f203d6bbd313427afb78fe163973a0/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b8c48d07-5599-4a74-94d4-d6488bde2827/volumes/kubernetes.io~projected/kube-api-access-qmnmx DeviceMajor:0 DeviceMinor:372 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:464 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-495 DeviceMajor:0 DeviceMinor:495 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-538 DeviceMajor:0 DeviceMinor:538 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:00ed8e3406e02fb MacAddress:d6:1a:02:35:ce:90 Speed:10000 Mtu:8901} {Name:010e66bf2b3c5d5 MacAddress:96:05:a1:78:b8:4d Speed:10000 Mtu:8901} {Name:2595dae84d252ab MacAddress:6e:88:48:cf:d2:7e Speed:10000 Mtu:8901} {Name:2cd2a81a83b1742 MacAddress:2a:85:85:86:b4:4d Speed:10000 Mtu:8901} {Name:4fa4df476257e40 MacAddress:7a:cc:13:a7:12:b6 Speed:10000 Mtu:8901} {Name:53f1621289f5ad3 MacAddress:02:76:c7:4a:b2:0b Speed:10000 Mtu:8901} {Name:544c576d92b15de MacAddress:4e:2a:76:9b:2d:b8 Speed:10000 Mtu:8901} {Name:7bab66348e40830 MacAddress:d2:53:b0:70:35:de Speed:10000 Mtu:8901} {Name:81b266645f022c9 MacAddress:22:59:bc:c1:5c:42 Speed:10000 Mtu:8901} {Name:9070bded93cff12 MacAddress:9a:63:0a:4c:ce:23 Speed:10000 Mtu:8901} {Name:a2f3cc0b7891f93 MacAddress:3e:69:b6:84:89:f4 Speed:10000 Mtu:8901} {Name:ad53fa1332f37c8 MacAddress:16:7f:ec:1e:7c:72 Speed:10000 Mtu:8901} {Name:bbdeeec904dc072 MacAddress:b2:65:b7:e3:e8:cd Speed:10000 Mtu:8901} {Name:bc17043616d9ef8 MacAddress:66:a8:64:4a:6d:0b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ff:0e:2b:a5 Speed:0 Mtu:9001} {Name:br-int MacAddress:76:81:b2:0f:b8:b9 Speed:0 Mtu:8901} {Name:d9ad9b60e9424a5 MacAddress:9e:bf:03:b2:07:c3 Speed:10000 Mtu:8901} {Name:e0abdd59e265409 MacAddress:f2:8e:8b:c4:27:d8 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ff:0e:2b:a5 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:4e:4a:10:8d:80:0a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:8a:be:42:0f:3f:1b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 00:35:06.202734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.202340    6099 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 00:35:06.202734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.202383    6099 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 00:35:06.202851 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.202748    6099 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 00:35:06.202946 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.202778    6099 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-245.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 00:35:06.203027 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.202949    6099 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 00:35:06.203027 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.202962    6099 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 00:35:06.203027 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.202977    6099 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:35:06.203027 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.203003    6099 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:35:06.203189 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.203096    6099 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 00:35:06.203307 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.203294    6099 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 00:35:06.203423 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.203412    6099 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 00:35:06.203483 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.203457    6099 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 00:35:06.203483 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.203477    6099 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 00:35:06.203572 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.203491    6099 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 00:35:06.203572 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.203503    6099 kubelet.go:405] "Adding apiserver pod source"
Oct 30 00:35:06.203572 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.203515    6099 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 00:35:06.204315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.204299    6099 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 00:35:06.204315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.204314    6099 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:35:06.204865 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.204848    6099 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 00:35:06.208311 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208295    6099 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 00:35:06.208311 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208309    6099 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:35:06.208409 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208402    6099 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 00:35:06.208590 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208579    6099 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 00:35:06.208621 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208608    6099 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 00:35:06.208713 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208693    6099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 00:35:06.208746 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208720    6099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 00:35:06.208746 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208727    6099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 00:35:06.208746 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208733    6099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 00:35:06.208746 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208738    6099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 00:35:06.208746 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208744    6099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 00:35:06.208878 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208750    6099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 00:35:06.208878 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208756    6099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 00:35:06.208878 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208762    6099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 00:35:06.208878 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208769    6099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 00:35:06.208878 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208777    6099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 00:35:06.208878 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208787    6099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 00:35:06.208878 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208812    6099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 00:35:06.208878 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.208817    6099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 00:35:06.212187 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.212172    6099 server.go:1263] "Started kubelet"
Oct 30 00:35:06.212288 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.212258    6099 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 00:35:06.212784 ip-10-0-130-245 systemd[1]: Started Kubernetes Kubelet.
Oct 30 00:35:06.213077 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.212288    6099 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 00:35:06.213552 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.212843    6099 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 00:35:06.213900 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.213882    6099 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 00:35:06.216642 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.216622    6099 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 00:35:06.219989 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.218947    6099 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 00:35:06.219989 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.218979    6099 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 00:35:06.219989 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.219013    6099 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 00:35:06.219989 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.219026    6099 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 00:35:06.219989 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.219048    6099 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:28:57 +0000 UTC" deadline="2028-07-08 02:33:32.184955472 +0000 UTC"
Oct 30 00:35:06.219989 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.219072    6099 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14809h58m25.965886574s"
Oct 30 00:35:06.219989 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:06.219104    6099 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found"
Oct 30 00:35:06.219989 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.219145    6099 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 00:35:06.219989 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.219912    6099 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 00:35:06.219989 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.219927    6099 factory.go:55] Registering systemd factory
Oct 30 00:35:06.219989 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.219939    6099 factory.go:223] Registration of the systemd container factory successfully
Oct 30 00:35:06.220572 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.220555    6099 factory.go:153] Registering CRI-O factory
Oct 30 00:35:06.220628 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.220577    6099 factory.go:223] Registration of the crio container factory successfully
Oct 30 00:35:06.220628 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.220596    6099 factory.go:103] Registering Raw factory
Oct 30 00:35:06.220628 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.220611    6099 manager.go:1199] Started watching for new ooms in manager
Oct 30 00:35:06.221010 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.220996    6099 manager.go:319] Starting recovery of all containers
Oct 30 00:35:06.222490 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222449    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53ec687-40dd-4358-bc05-83d064d9d082" volumeName="kubernetes.io/empty-dir/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-textfile" seLinuxMountContext=""
Oct 30 00:35:06.222593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222490    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-config-out" seLinuxMountContext=""
Oct 30 00:35:06.222593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222505    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-metrics-client-certs" seLinuxMountContext=""
Oct 30 00:35:06.222593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222518    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="070879ac-09e8-4fdb-b55f-e52a9998552f" volumeName="kubernetes.io/projected/070879ac-09e8-4fdb-b55f-e52a9998552f-kube-api-access-257nk" seLinuxMountContext=""
Oct 30 00:35:06.222593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222529    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70c3a600-551a-4ff4-8fbe-e9a2aa1172e3" volumeName="kubernetes.io/configmap/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-config-volume" seLinuxMountContext=""
Oct 30 00:35:06.222593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222541    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="962a4e53-88b6-4866-abc5-108b6d5ce784" volumeName="kubernetes.io/secret/962a4e53-88b6-4866-abc5-108b6d5ce784-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 30 00:35:06.222593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222552    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc8e4d97-4c7f-450d-961c-528c992a29b2" volumeName="kubernetes.io/configmap/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 30 00:35:06.222593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222563    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" volumeName="kubernetes.io/empty-dir/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-tmp" seLinuxMountContext=""
Oct 30 00:35:06.222593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222577    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8c48d07-5599-4a74-94d4-d6488bde2827" volumeName="kubernetes.io/configmap/b8c48d07-5599-4a74-94d4-d6488bde2827-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 00:35:06.222593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222589    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53ec687-40dd-4358-bc05-83d064d9d082" volumeName="kubernetes.io/projected/f53ec687-40dd-4358-bc05-83d064d9d082-kube-api-access-l6gmn" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222600    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-kube-api-access-j5kb7" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222611    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="070879ac-09e8-4fdb-b55f-e52a9998552f" volumeName="kubernetes.io/configmap/070879ac-09e8-4fdb-b55f-e52a9998552f-env-overrides" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222623    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426a04b3-dc76-4b89-8d28-3038652fd7bf" volumeName="kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-tls-assets" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222635    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="733527bc-83b7-423c-a79e-1de279f5a607" volumeName="kubernetes.io/projected/733527bc-83b7-423c-a79e-1de279f5a607-kube-api-access-jhvbj" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222650    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5282344-9beb-485c-9525-6b09bdda0804" volumeName="kubernetes.io/projected/c5282344-9beb-485c-9525-6b09bdda0804-kube-api-access-wpw5w" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222662    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5282344-9beb-485c-9525-6b09bdda0804" volumeName="kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222673    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222685    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426a04b3-dc76-4b89-8d28-3038652fd7bf" volumeName="kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222695    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc8e4d97-4c7f-450d-961c-528c992a29b2" volumeName="kubernetes.io/empty-dir/bc8e4d97-4c7f-450d-961c-528c992a29b2-volume-directive-shadow" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222720    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc8e4d97-4c7f-450d-961c-528c992a29b2" volumeName="kubernetes.io/secret/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222733    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddb2dce5-50be-42dc-88f4-ecb4d6a29d28" volumeName="kubernetes.io/configmap/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-metrics-client-ca" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222758    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6" volumeName="kubernetes.io/empty-dir/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-data-volume" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222769    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6" volumeName="kubernetes.io/projected/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-kube-api-access-xzmdn" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222788    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32086945-d23d-49bb-a876-fab4cef52ec6" volumeName="kubernetes.io/configmap/32086945-d23d-49bb-a876-fab4cef52ec6-iptables-alerter-script" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222800    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac72880-98ab-4093-b7aa-903a37c8113e" volumeName="kubernetes.io/configmap/5ac72880-98ab-4093-b7aa-903a37c8113e-cni-binary-copy" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222811    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="962a4e53-88b6-4866-abc5-108b6d5ce784" volumeName="kubernetes.io/configmap/962a4e53-88b6-4866-abc5-108b6d5ce784-metrics-client-ca" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222862    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5282344-9beb-485c-9525-6b09bdda0804" volumeName="kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222881    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddb2dce5-50be-42dc-88f4-ecb4d6a29d28" volumeName="kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.222994    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8c48d07-5599-4a74-94d4-d6488bde2827" volumeName="kubernetes.io/secret/b8c48d07-5599-4a74-94d4-d6488bde2827-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223021    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426a04b3-dc76-4b89-8d28-3038652fd7bf" volumeName="kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-web-config" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223034    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="070879ac-09e8-4fdb-b55f-e52a9998552f" volumeName="kubernetes.io/configmap/070879ac-09e8-4fdb-b55f-e52a9998552f-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223045    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223057    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426a04b3-dc76-4b89-8d28-3038652fd7bf" volumeName="kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223089    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b45c13d7-29c0-4054-9b7d-fd33ecec9cf7" volumeName="kubernetes.io/empty-dir/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7-tmp-dir" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223102    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-tls-assets" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223113    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5975a832-beb3-4e76-98ad-aef36a2ccce2" volumeName="kubernetes.io/projected/5975a832-beb3-4e76-98ad-aef36a2ccce2-kube-api-access-cnd4w" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223125    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21be6874-6138-4a00-9c6e-aa1841eb3230" volumeName="kubernetes.io/secret/21be6874-6138-4a00-9c6e-aa1841eb3230-prometheus-operator-tls" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223138    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5282344-9beb-485c-9525-6b09bdda0804" volumeName="kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-grpc-tls" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223150    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddb2dce5-50be-42dc-88f4-ecb4d6a29d28" volumeName="kubernetes.io/configmap/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223177    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223197    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70c3a600-551a-4ff4-8fbe-e9a2aa1172e3" volumeName="kubernetes.io/projected/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-kube-api-access-fpbnj" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223209    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8c48d07-5599-4a74-94d4-d6488bde2827" volumeName="kubernetes.io/secret/b8c48d07-5599-4a74-94d4-d6488bde2827-client-ca-bundle" seLinuxMountContext=""
Oct 30 00:35:06.223216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223222    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d996f136-ba95-4636-a870-94520d3eff0f" volumeName="kubernetes.io/configmap/d996f136-ba95-4636-a870-94520d3eff0f-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223237    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-db" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223248    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" volumeName="kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223261    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47d34735-e4b1-4073-b6de-98a9a720432b" volumeName="kubernetes.io/secret/47d34735-e4b1-4073-b6de-98a9a720432b-agent-certs" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223272    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8c48d07-5599-4a74-94d4-d6488bde2827" volumeName="kubernetes.io/configmap/b8c48d07-5599-4a74-94d4-d6488bde2827-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223283    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddb2dce5-50be-42dc-88f4-ecb4d6a29d28" volumeName="kubernetes.io/projected/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-kube-api-access-9ncsp" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223295    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426a04b3-dc76-4b89-8d28-3038652fd7bf" volumeName="kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-main-db" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223305    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5975a832-beb3-4e76-98ad-aef36a2ccce2" volumeName="kubernetes.io/configmap/5975a832-beb3-4e76-98ad-aef36a2ccce2-serviceca" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223317    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" volumeName="kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223330    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21be6874-6138-4a00-9c6e-aa1841eb3230" volumeName="kubernetes.io/secret/21be6874-6138-4a00-9c6e-aa1841eb3230-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223342    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b73f4c46-cb05-492e-8dbc-953f3c70b1d5" volumeName="kubernetes.io/projected/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-kube-api-access-f24vm" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223355    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5282344-9beb-485c-9525-6b09bdda0804" volumeName="kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223366    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b308ab3-9eb7-4be8-964f-ff94af40b883" volumeName="kubernetes.io/secret/0b308ab3-9eb7-4be8-964f-ff94af40b883-cert" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223385    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-web-config" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223407    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32086945-d23d-49bb-a876-fab4cef52ec6" volumeName="kubernetes.io/projected/32086945-d23d-49bb-a876-fab4cef52ec6-kube-api-access-d9rmw" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223419    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426a04b3-dc76-4b89-8d28-3038652fd7bf" volumeName="kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223431    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="962a4e53-88b6-4866-abc5-108b6d5ce784" volumeName="kubernetes.io/projected/962a4e53-88b6-4866-abc5-108b6d5ce784-kube-api-access-lhjkw" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223443    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8c48d07-5599-4a74-94d4-d6488bde2827" volumeName="kubernetes.io/projected/b8c48d07-5599-4a74-94d4-d6488bde2827-kube-api-access-qmnmx" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223454    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc8e4d97-4c7f-450d-961c-528c992a29b2" volumeName="kubernetes.io/secret/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-state-metrics-tls" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223466    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5282344-9beb-485c-9525-6b09bdda0804" volumeName="kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223477    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="070879ac-09e8-4fdb-b55f-e52a9998552f" volumeName="kubernetes.io/secret/070879ac-09e8-4fdb-b55f-e52a9998552f-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223500    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d996f136-ba95-4636-a870-94520d3eff0f" volumeName="kubernetes.io/projected/d996f136-ba95-4636-a870-94520d3eff0f-kube-api-access-vbtjm" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223511    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddb2dce5-50be-42dc-88f4-ecb4d6a29d28" volumeName="kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-secret-telemeter-client" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223522    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53ec687-40dd-4358-bc05-83d064d9d082" volumeName="kubernetes.io/configmap/f53ec687-40dd-4358-bc05-83d064d9d082-metrics-client-ca" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223536    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53ec687-40dd-4358-bc05-83d064d9d082" volumeName="kubernetes.io/configmap/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223546    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426a04b3-dc76-4b89-8d28-3038652fd7bf" volumeName="kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-kube-api-access-fwm8g" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223559    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" volumeName="kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223572    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12bf7b6a-fd62-4921-910d-f9fff2b347d1" volumeName="kubernetes.io/secret/12bf7b6a-fd62-4921-910d-f9fff2b347d1-monitoring-plugin-cert" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223583    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426a04b3-dc76-4b89-8d28-3038652fd7bf" volumeName="kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-volume" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223596    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddb2dce5-50be-42dc-88f4-ecb4d6a29d28" volumeName="kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-federate-client-tls" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223608    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="070879ac-09e8-4fdb-b55f-e52a9998552f" volumeName="kubernetes.io/configmap/070879ac-09e8-4fdb-b55f-e52a9998552f-ovnkube-config" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223620    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223633    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223645    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426a04b3-dc76-4b89-8d28-3038652fd7bf" volumeName="kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-out" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223657    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70c3a600-551a-4ff4-8fbe-e9a2aa1172e3" volumeName="kubernetes.io/secret/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-metrics-tls" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223669    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddb2dce5-50be-42dc-88f4-ecb4d6a29d28" volumeName="kubernetes.io/configmap/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223681    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddb2dce5-50be-42dc-88f4-ecb4d6a29d28" volumeName="kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-telemeter-client-tls" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223692    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223719    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223732    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-config" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223743    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426a04b3-dc76-4b89-8d28-3038652fd7bf" volumeName="kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-metrics-client-ca" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223755    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426a04b3-dc76-4b89-8d28-3038652fd7bf" volumeName="kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223769    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426a04b3-dc76-4b89-8d28-3038652fd7bf" volumeName="kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223781    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b73f4c46-cb05-492e-8dbc-953f3c70b1d5" volumeName="kubernetes.io/empty-dir/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-tmp" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223797    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b73f4c46-cb05-492e-8dbc-953f3c70b1d5" volumeName="kubernetes.io/empty-dir/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-tuned" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223820    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8c48d07-5599-4a74-94d4-d6488bde2827" volumeName="kubernetes.io/secret/b8c48d07-5599-4a74-94d4-d6488bde2827-secret-metrics-server-tls" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223831    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" volumeName="kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223844    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d996f136-ba95-4636-a870-94520d3eff0f" volumeName="kubernetes.io/configmap/d996f136-ba95-4636-a870-94520d3eff0f-cni-binary-copy" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223856    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223868    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac72880-98ab-4093-b7aa-903a37c8113e" volumeName="kubernetes.io/projected/5ac72880-98ab-4093-b7aa-903a37c8113e-kube-api-access-bmqr7" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223879    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8c48d07-5599-4a74-94d4-d6488bde2827" volumeName="kubernetes.io/empty-dir/b8c48d07-5599-4a74-94d4-d6488bde2827-audit-log" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223892    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21be6874-6138-4a00-9c6e-aa1841eb3230" volumeName="kubernetes.io/configmap/21be6874-6138-4a00-9c6e-aa1841eb3230-metrics-client-ca" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223903    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70c3a600-551a-4ff4-8fbe-e9a2aa1172e3" volumeName="kubernetes.io/empty-dir/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-tmp-dir" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223915    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9803898-12ec-4a07-9a8f-1eef3f89235b" volumeName="kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223928    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53ec687-40dd-4358-bc05-83d064d9d082" volumeName="kubernetes.io/secret/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223940    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" volumeName="kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223951    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223965    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223976    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b308ab3-9eb7-4be8-964f-ff94af40b883" volumeName="kubernetes.io/projected/0b308ab3-9eb7-4be8-964f-ff94af40b883-kube-api-access-6j86q" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.223987    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6" volumeName="kubernetes.io/configmap/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224008    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47d34735-e4b1-4073-b6de-98a9a720432b" volumeName="kubernetes.io/configmap/47d34735-e4b1-4073-b6de-98a9a720432b-konnectivity-ca" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224031    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5282344-9beb-485c-9525-6b09bdda0804" volumeName="kubernetes.io/configmap/c5282344-9beb-485c-9525-6b09bdda0804-metrics-client-ca" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224043    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a145ece-ad48-43c2-a259-b9590802cfe8" volumeName="kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224055    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7f00b4a-044e-49ab-8f43-4351590692e8" volumeName="kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224067    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12bf7b6a-fd62-4921-910d-f9fff2b347d1" volumeName="kubernetes.io/projected/12bf7b6a-fd62-4921-910d-f9fff2b347d1-kube-api-access-j7spz" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224079    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-grpc-tls" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224091    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224102    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426a04b3-dc76-4b89-8d28-3038652fd7bf" volumeName="kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-cluster-tls-config" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224114    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5282344-9beb-485c-9525-6b09bdda0804" volumeName="kubernetes.io/empty-dir/c5282344-9beb-485c-9525-6b09bdda0804-tmp" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224126    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d996f136-ba95-4636-a870-94520d3eff0f" volumeName="kubernetes.io/configmap/d996f136-ba95-4636-a870-94520d3eff0f-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224157    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21be6874-6138-4a00-9c6e-aa1841eb3230" volumeName="kubernetes.io/projected/21be6874-6138-4a00-9c6e-aa1841eb3230-kube-api-access-p7lkm" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224169    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a145ece-ad48-43c2-a259-b9590802cfe8" volumeName="kubernetes.io/projected/3a145ece-ad48-43c2-a259-b9590802cfe8-kube-api-access-8nbn8" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224181    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac72880-98ab-4093-b7aa-903a37c8113e" volumeName="kubernetes.io/configmap/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-daemon-config" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224193    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5282344-9beb-485c-9525-6b09bdda0804" volumeName="kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224205    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53ec687-40dd-4358-bc05-83d064d9d082" volumeName="kubernetes.io/secret/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-tls" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224218    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" volumeName="kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224233    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b45c13d7-29c0-4054-9b7d-fd33ecec9cf7" volumeName="kubernetes.io/projected/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7-kube-api-access-f4tjg" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224245    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc8e4d97-4c7f-450d-961c-528c992a29b2" volumeName="kubernetes.io/projected/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-api-access-mkp59" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224257    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026218e5-d0ac-48b3-9023-f245d5cf9848" volumeName="kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224268    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6" volumeName="kubernetes.io/secret/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224287    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="962a4e53-88b6-4866-abc5-108b6d5ce784" volumeName="kubernetes.io/secret/962a4e53-88b6-4866-abc5-108b6d5ce784-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224300    6099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc8e4d97-4c7f-450d-961c-528c992a29b2" volumeName="kubernetes.io/configmap/bc8e4d97-4c7f-450d-961c-528c992a29b2-metrics-client-ca" seLinuxMountContext=""
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224312    6099 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 00:35:06.224780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.224320    6099 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 00:35:06.251697 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.251649    6099 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 00:35:06.253315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.253289    6099 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 00:35:06.253315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.253313    6099 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 00:35:06.253502 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.253333    6099 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 00:35:06.253502 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:06.253374    6099 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 00:35:06.255066 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.255001    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 00:35:06.255977 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.255954    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 00:35:06.259471 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.259450    6099 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 00:35:06.259674 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.259655    6099 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:35:06.268624 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.268599    6099 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:35:06.274862 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.274841    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 00:35:06.275227 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.275204    6099 generic.go:356] "Generic (PLEG): container finished" podID="070879ac-09e8-4fdb-b55f-e52a9998552f" containerID="59428dffec1172c34500f519af0fb69b41510655bbd9b5635cb46d0a5324ac60" exitCode=1
Oct 30 00:35:06.278948 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.278926    6099 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:35:06.279383 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.279364    6099 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:35:06.286340 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.286316    6099 generic.go:356] "Generic (PLEG): container finished" podID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerID="5be1dff3726a9bbaf208d7e7c745e2af4e1fb89d65e25c3dfb3564f5a9011269" exitCode=0
Oct 30 00:35:06.290199 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.290174    6099 generic.go:356] "Generic (PLEG): container finished" podID="f53ec687-40dd-4358-bc05-83d064d9d082" containerID="742019ae4ad77df39ea4746b0a561d5778e19c03fb8a5df83a90532a0c62b4ef" exitCode=0
Oct 30 00:35:06.305359 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.305334    6099 generic.go:356] "Generic (PLEG): container finished" podID="553cfdac89599e545ab5a9539f3d3b39" containerID="00c9adad48ce7102684c8f7be5cc6c6245f7ff1de066b0ff9cc77dcf355bc510" exitCode=0
Oct 30 00:35:06.317202 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.317181    6099 generic.go:356] "Generic (PLEG): container finished" podID="d996f136-ba95-4636-a870-94520d3eff0f" containerID="72be51cb9ed7eff16bc20d3f12939abdace8727c78b193f4dda082cb5469776f" exitCode=0
Oct 30 00:35:06.317202 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.317202    6099 generic.go:356] "Generic (PLEG): container finished" podID="d996f136-ba95-4636-a870-94520d3eff0f" containerID="8e83bb8f0891b297a06d182c99dec138c964d7fba95fefe6859c771bbd0dc4e9" exitCode=0
Oct 30 00:35:06.317358 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.317227    6099 generic.go:356] "Generic (PLEG): container finished" podID="d996f136-ba95-4636-a870-94520d3eff0f" containerID="e73e2077a4c8792ae759ea515baf233207dba26ab14d2f08a27dadfdf6f05003" exitCode=0
Oct 30 00:35:06.317358 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.317236    6099 generic.go:356] "Generic (PLEG): container finished" podID="d996f136-ba95-4636-a870-94520d3eff0f" containerID="8205418c25c455ff29f3f6f0916f344b493f31b5f5207986d8742398e359fe58" exitCode=0
Oct 30 00:35:06.317358 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.317247    6099 generic.go:356] "Generic (PLEG): container finished" podID="d996f136-ba95-4636-a870-94520d3eff0f" containerID="8ad112f314388e3afc623fc4b39d7bd8de911433551c542a1dd52b61f5dc9ce7" exitCode=0
Oct 30 00:35:06.317358 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.317255    6099 generic.go:356] "Generic (PLEG): container finished" podID="d996f136-ba95-4636-a870-94520d3eff0f" containerID="56a19c900ea14a288c773079a23d990e8ea27c13237434de39878047dea33c73" exitCode=0
Oct 30 00:35:06.326107 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.326086    6099 generic.go:356] "Generic (PLEG): container finished" podID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerID="289fa18bc40491e3b05eba723fa2e20720972feb42138c2f8902d6a3f206dc45" exitCode=0
Oct 30 00:35:06.353488 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:06.353435    6099 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 00:35:06.358113 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.358098    6099 manager.go:324] Recovery completed
Oct 30 00:35:06.386428 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.386407    6099 cpu_manager.go:226] "Starting" policy="none"
Oct 30 00:35:06.386428 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.386426    6099 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 00:35:06.386530 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.386447    6099 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 00:35:06.386607 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.386590    6099 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 00:35:06.386664 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.386609    6099 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 00:35:06.386664 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.386632    6099 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 00:35:06.386664 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.386640    6099 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 00:35:06.386664 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.386649    6099 policy_none.go:50] "Start"
Oct 30 00:35:06.386664 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.386660    6099 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 00:35:06.386853 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.386671    6099 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 00:35:06.386853 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.386821    6099 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 00:35:06.386853 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.386828    6099 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 00:35:06.386853 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.386833    6099 policy_none.go:44] "Start"
Oct 30 00:35:06.391632 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.391603    6099 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 00:35:06.394229 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.394215    6099 manager.go:341] "Starting Device Plugin manager"
Oct 30 00:35:06.394858 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:06.394240    6099 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 00:35:06.394915 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.394860    6099 server.go:87] "Starting device plugin registration server"
Oct 30 00:35:06.395103 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.395093    6099 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 00:35:06.395133 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.395106    6099 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 00:35:06.395215 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.395192    6099 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 00:35:06.395915 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.395874    6099 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 00:35:06.395998 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.395926    6099 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 00:35:06.395998 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.395934    6099 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 00:35:06.396061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.395987    6099 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:35:06.395911799Z","UUID":"377efaa6-750e-4b75-b159-5c9cb70c2317","Handler":null,"Name":"","Endpoint":""}
Oct 30 00:35:06.397338 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.397322    6099 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 00:35:06.397431 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.397381    6099 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 00:35:06.397486 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:06.397378    6099 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 00:35:06.495586 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.495553    6099 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:35:06.497480 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.497464    6099 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:35:06.497582 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.497493    6099 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:35:06.497582 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.497503    6099 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:35:06.497697 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.497586    6099 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-245.ec2.internal"
Oct 30 00:35:06.527152 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.527128    6099 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-245.ec2.internal"
Oct 30 00:35:06.527250 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.527192    6099 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-245.ec2.internal"
Oct 30 00:35:06.554190 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.554128    6099 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal"]
Oct 30 00:35:06.554827 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.554804    6099 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:35:06.554933 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.554825    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" event={"ID":"553cfdac89599e545ab5a9539f3d3b39","Type":"ContainerStarted","Data":"7fc5fe31094c3b41c9d6235f865d53d0107e10e86778040607d24abbfa8a9908"}
Oct 30 00:35:06.554933 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.554860    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" event={"ID":"553cfdac89599e545ab5a9539f3d3b39","Type":"ContainerDied","Data":"00c9adad48ce7102684c8f7be5cc6c6245f7ff1de066b0ff9cc77dcf355bc510"}
Oct 30 00:35:06.554933 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.554874    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" event={"ID":"553cfdac89599e545ab5a9539f3d3b39","Type":"ContainerStarted","Data":"edaebab344eb8b504f4cbb62b8b245ce7e6a502dfbe2011df04d7030ddd7e6e8"}
Oct 30 00:35:06.554933 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.554913    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal" event={"ID":"f8c3a9edbd8cbf03175beab8c4db7242","Type":"ContainerStarted","Data":"1e5b061c695be95293eb2b31260a52d9a68b4a5c469a4ef5870b2e8bd8074406"}
Oct 30 00:35:06.554933 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.554933    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal" event={"ID":"f8c3a9edbd8cbf03175beab8c4db7242","Type":"ContainerStarted","Data":"ab0c9e3bbd550a22c57907f05402ac9314905c0c759dec1a19a3abf77b32903e"}
Oct 30 00:35:06.555109 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.554961    6099 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal"
Oct 30 00:35:06.580498 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.580477    6099 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:35:06.580498 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.580495    6099 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:35:06.580643 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:06.580528    6099 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-245.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal"
Oct 30 00:35:06.580643 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:06.580533    6099 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:35:06.660939 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.660867    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/553cfdac89599e545ab5a9539f3d3b39-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" (UID: \"553cfdac89599e545ab5a9539f3d3b39\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:35:06.660939 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.660899    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/553cfdac89599e545ab5a9539f3d3b39-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" (UID: \"553cfdac89599e545ab5a9539f3d3b39\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:35:06.660939 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.660917    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f8c3a9edbd8cbf03175beab8c4db7242-config\") pod \"kube-apiserver-proxy-ip-10-0-130-245.ec2.internal\" (UID: \"f8c3a9edbd8cbf03175beab8c4db7242\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal"
Oct 30 00:35:06.761406 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.761366    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/553cfdac89599e545ab5a9539f3d3b39-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" (UID: \"553cfdac89599e545ab5a9539f3d3b39\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:35:06.761548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.761417    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/553cfdac89599e545ab5a9539f3d3b39-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" (UID: \"553cfdac89599e545ab5a9539f3d3b39\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:35:06.761548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.761437    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f8c3a9edbd8cbf03175beab8c4db7242-config\") pod \"kube-apiserver-proxy-ip-10-0-130-245.ec2.internal\" (UID: \"f8c3a9edbd8cbf03175beab8c4db7242\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal"
Oct 30 00:35:06.761548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.761440    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/553cfdac89599e545ab5a9539f3d3b39-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" (UID: \"553cfdac89599e545ab5a9539f3d3b39\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:35:06.761548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.761476    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f8c3a9edbd8cbf03175beab8c4db7242-config\") pod \"kube-apiserver-proxy-ip-10-0-130-245.ec2.internal\" (UID: \"f8c3a9edbd8cbf03175beab8c4db7242\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal"
Oct 30 00:35:06.761548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:06.761513    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/553cfdac89599e545ab5a9539f3d3b39-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" (UID: \"553cfdac89599e545ab5a9539f3d3b39\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:35:07.203940 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.203909    6099 apiserver.go:52] "Watching apiserver"
Oct 30 00:35:07.248497 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.248463    6099 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 00:35:07.250628 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.250584    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-vcqkc","openshift-cluster-node-tuning-operator/tuned-9h5hq","openshift-monitoring/telemeter-client-6ddf586968-ml4vv","openshift-dns/node-resolver-b2bnk","openshift-monitoring/monitoring-plugin-77f4f76594-tljgj","openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh","kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal","openshift-image-registry/node-ca-d9lhd","openshift-monitoring/node-exporter-wb627","openshift-monitoring/thanos-querier-676949fd44-65zsj","kube-system/global-pull-secret-syncer-7vsmg","openshift-console/console-57449f4d45-vtkst","openshift-ingress-canary/ingress-canary-xfk62","openshift-insights/insights-runtime-extractor-wvmfk","openshift-monitoring/prometheus-k8s-0","openshift-multus/multus-42gzj","openshift-multus/multus-additional-cni-plugins-gmgzr","openshift-multus/network-metrics-daemon-wb6st","openshift-network-operator/iptables-alerter-whx68","openshift-dns/dns-default-rwwwf","openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz","openshift-monitoring/alertmanager-main-0","openshift-monitoring/metrics-server-7c55cf7f78-h65h8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal","openshift-network-diagnostics/network-check-target-62gpq","kube-system/konnectivity-agent-v6rdh","openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"]
Oct 30 00:35:07.264126 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.264085    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-x59g5\""
Oct 30 00:35:07.264645 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.264619    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-78zxw\""
Oct 30 00:35:07.265164 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.265140    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 00:35:07.265504 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.265467    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 00:35:07.265632 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.265611    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 00:35:07.265722 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.265599    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 00:35:07.265722 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.265595    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 00:35:07.266603 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.266576    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 00:35:07.267175 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.267153    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 00:35:07.267260 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.267236    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 00:35:07.267420 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.267403    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jk5pr\""
Oct 30 00:35:07.267542 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.267524    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 00:35:07.267754 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.267701    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 00:35:07.267855 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.267831    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 00:35:07.268171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.268148    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q2pzq\""
Oct 30 00:35:07.268247 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.268204    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 00:35:07.268529 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.268511    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:35:07.268597 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.268529    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 00:35:07.269229 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.269206    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 00:35:07.269322 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.269311    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qpxdw\""
Oct 30 00:35:07.269512 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.269494    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 00:35:07.269840 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.269823    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 00:35:07.269930 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.269915    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b4svw\""
Oct 30 00:35:07.270105 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.270091    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 00:35:07.272133 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.272108    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 00:35:07.272368 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.272190    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 00:35:07.272368 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.272347    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 00:35:07.272505 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.272373    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-smldl\""
Oct 30 00:35:07.272505 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.272375    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:35:07.272650 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.272584    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 00:35:07.272650 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.272643    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 00:35:07.272804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.272754    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 00:35:07.272862 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.272830    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 00:35:07.273323 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.273308    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 00:35:07.273497 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.273378    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 00:35:07.273497 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.273458    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 00:35:07.273932 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.273671    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 00:35:07.274775 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.274755    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 00:35:07.275164 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.275142    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c889x\""
Oct 30 00:35:07.275164 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.275148    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 00:35:07.275277 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.275164    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 00:35:07.275277 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.275225    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 00:35:07.275414 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.275392    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 00:35:07.275561 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.275529    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-75g57\""
Oct 30 00:35:07.275561 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.275548    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q972j\""
Oct 30 00:35:07.275781 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.275583    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 00:35:07.275781 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.275594    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 00:35:07.275781 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.275630    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 00:35:07.275781 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.275651    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 00:35:07.275781 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.275687    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 00:35:07.275781 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.275690    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 00:35:07.276048 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.275800    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 00:35:07.276048 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.275857    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 00:35:07.276048 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.275968    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-s9968\""
Oct 30 00:35:07.276048 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.275976    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 00:35:07.276048 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.275995    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 00:35:07.276048 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.276017    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 00:35:07.276278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.276059    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pkqjh\""
Oct 30 00:35:07.276278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.276100    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ltmbf\""
Oct 30 00:35:07.276278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.276112    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xjltb\""
Oct 30 00:35:07.276278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.276131    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 00:35:07.276278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.276146    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tcw96\""
Oct 30 00:35:07.276278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.276149    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 00:35:07.276278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.276163    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x9vrg\""
Oct 30 00:35:07.276278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.276226    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 00:35:07.276597 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.276285    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mkc6n\""
Oct 30 00:35:07.276597 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.276334    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 00:35:07.276597 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.276441    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 00:35:07.276597 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.276491    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qff2g\""
Oct 30 00:35:07.276597 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.276578    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 00:35:07.277473 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.277457    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 00:35:07.322440 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.322418    6099 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 00:35:07.330761 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.330744    6099 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal"
Oct 30 00:35:07.330879 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.330866    6099 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:35:07.353827 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.353805    6099 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:35:07.353906 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.353852    6099 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-245.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal"
Oct 30 00:35:07.354328 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.354314    6099 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:35:07.354400 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.354352    6099 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal"
Oct 30 00:35:07.363643 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.363623    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-kubernetes\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.363733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.363649    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-metrics-client-ca\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:35:07.363733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.363667    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzmdn\" (UniqueName: \"kubernetes.io/projected/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-kube-api-access-xzmdn\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:35:07.363733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.363685    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-system-cni-dir\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.363733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.363702    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-device-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.363883 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.363765    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-run-ovn\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.363883 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.363793    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-serving-certs-ca-bundle\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:35:07.363883 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.363821    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:35:07.363883 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.363840    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-data-volume\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:35:07.363883 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.363857    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:35:07.363883 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.363875    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bc8e4d97-4c7f-450d-961c-528c992a29b2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:35:07.364124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364003    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21be6874-6138-4a00-9c6e-aa1841eb3230-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:35:07.364124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364050    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-data-volume\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:35:07.364124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364059    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bc8e4d97-4c7f-450d-961c-528c992a29b2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:35:07.364124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364052    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7spz\" (UniqueName: \"kubernetes.io/projected/12bf7b6a-fd62-4921-910d-f9fff2b347d1-kube-api-access-j7spz\") pod \"monitoring-plugin-77f4f76594-tljgj\" (UID: \"12bf7b6a-fd62-4921-910d-f9fff2b347d1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj"
Oct 30 00:35:07.364124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364106    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:35:07.364124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364117    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:35:07.364421 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364132    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-metrics-client-ca\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:35:07.364421 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364218    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21be6874-6138-4a00-9c6e-aa1841eb3230-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:35:07.364421 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364410    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b308ab3-9eb7-4be8-964f-ff94af40b883-cert\") pod \"ingress-canary-xfk62\" (UID: \"0b308ab3-9eb7-4be8-964f-ff94af40b883\") " pod="openshift-ingress-canary/ingress-canary-xfk62"
Oct 30 00:35:07.364421 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364418    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 00:35:07.364553 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364437    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:07.364553 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364490    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:07.364553 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364517    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5282344-9beb-485c-9525-6b09bdda0804-tmp\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:07.364553 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364534    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b7f00b4a-044e-49ab-8f43-4351590692e8-original-pull-secret\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:35:07.364747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364540    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b8c48d07-5599-4a74-94d4-d6488bde2827-audit-log\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:35:07.364747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364586    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21be6874-6138-4a00-9c6e-aa1841eb3230-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:35:07.364747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364611    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b8c48d07-5599-4a74-94d4-d6488bde2827-audit-log\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:35:07.364747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364615    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhjkw\" (UniqueName: \"kubernetes.io/projected/962a4e53-88b6-4866-abc5-108b6d5ce784-kube-api-access-lhjkw\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:35:07.364747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364635    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b308ab3-9eb7-4be8-964f-ff94af40b883-cert\") pod \"ingress-canary-xfk62\" (UID: \"0b308ab3-9eb7-4be8-964f-ff94af40b883\") " pod="openshift-ingress-canary/ingress-canary-xfk62"
Oct 30 00:35:07.364747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364641    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:07.364747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364641    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5282344-9beb-485c-9525-6b09bdda0804-tmp\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:07.364747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364685    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 00:35:07.364747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364701    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 00:35:07.365172 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364851    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 00:35:07.365172 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364859    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 00:35:07.365172 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364881    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 00:35:07.365172 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.364966    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21be6874-6138-4a00-9c6e-aa1841eb3230-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:35:07.365172 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.365070    6099 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 00:35:07.365172 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365105    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkp59\" (UniqueName: \"kubernetes.io/projected/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-api-access-mkp59\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:35:07.365172 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.365136    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:07.865120528 +0000 UTC m=+1.722028291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 00:35:07.365172 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365164    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.365548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365192    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-sysconfig\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.365548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365315    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.365548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365374    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 00:35:07.365548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365391    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-kubelet\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.365548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365426    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-telemeter-client-tls\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:35:07.365548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365448    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 00:35:07.365548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365453    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 00:35:07.365548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365467    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b7f00b4a-044e-49ab-8f43-4351590692e8-dbus\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:35:07.365548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365465    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-r25c7\""
Oct 30 00:35:07.365548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365477    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 00:35:07.365548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365504    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 00:35:07.365548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365505    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 00:35:07.365548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365338    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 00:35:07.365548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365511    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-accelerators-collector-config\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.365548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365538    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365566    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365645    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365650    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365664    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365657    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365732    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365753    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-socket-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365776    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365795    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365800    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365831    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32086945-d23d-49bb-a876-fab4cef52ec6-iptables-alerter-script\") pod \"iptables-alerter-whx68\" (UID: \"32086945-d23d-49bb-a876-fab4cef52ec6\") " pod="openshift-network-operator/iptables-alerter-whx68"
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365842    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-x2hxt\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365871    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365865    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-accelerators-collector-config\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365885    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365794    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365919    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365857    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f53ec687-40dd-4358-bc05-83d064d9d082-sys\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365930    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365856    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365805    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365962    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365963    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpw5w\" (UniqueName: \"kubernetes.io/projected/c5282344-9beb-485c-9525-6b09bdda0804-kube-api-access-wpw5w\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366005    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hxlc4\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366008    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366004    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-host\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365924    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366048    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.365834    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ks6f2\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366076    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366093    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f53ec687-40dd-4358-bc05-83d064d9d082-metrics-client-ca\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366117    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-78e6gfjg9uta9\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366153    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-var-lib-cni-multus\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366188    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366214    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7f0anuf4jt4pi\""
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366240    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32086945-d23d-49bb-a876-fab4cef52ec6-iptables-alerter-script\") pod \"iptables-alerter-whx68\" (UID: \"32086945-d23d-49bb-a876-fab4cef52ec6\") " pod="openshift-network-operator/iptables-alerter-whx68"
Oct 30 00:35:07.366245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366249    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366321    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f53ec687-40dd-4358-bc05-83d064d9d082-metrics-client-ca\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366336    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-82whn\""
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366359    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7-hosts-file\") pod \"node-resolver-b2bnk\" (UID: \"b45c13d7-29c0-4054-9b7d-fd33ecec9cf7\") " pod="openshift-dns/node-resolver-b2bnk"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366378    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mk2sw\""
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366380    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9qdg95h0bhori\""
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366417    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366423    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366418    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366461    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366466    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366467    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366490    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-etc-openvswitch\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366518    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-config-volume\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366557    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366568    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbtjm\" (UniqueName: \"kubernetes.io/projected/d996f136-ba95-4636-a870-94520d3eff0f-kube-api-access-vbtjm\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366592    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-web-config\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366624    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366648    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xzvlc\""
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366651    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-log-socket\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366678    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59hp\" (UniqueName: \"kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366690    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366701    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs\") pod \"network-metrics-daemon-wb6st\" (UID: \"3a145ece-ad48-43c2-a259-b9590802cfe8\") " pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366757    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-config-volume\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366778    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-run-netns\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366812    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhvbj\" (UniqueName: \"kubernetes.io/projected/733527bc-83b7-423c-a79e-1de279f5a607-kube-api-access-jhvbj\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366845    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366873    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-modprobe-d\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366883    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366898    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366950    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-run-systemd\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366975    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-run\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.366981    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-web-config\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367001    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/21be6874-6138-4a00-9c6e-aa1841eb3230-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367027    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12bf7b6a-fd62-4921-910d-f9fff2b347d1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-tljgj\" (UID: \"12bf7b6a-fd62-4921-910d-f9fff2b347d1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367071    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a145ece-ad48-43c2-a259-b9590802cfe8-metrics-certs\") pod \"network-metrics-daemon-wb6st\" (UID: \"3a145ece-ad48-43c2-a259-b9590802cfe8\") " pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367086    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-systemd-units\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367106    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367215    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367255    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12bf7b6a-fd62-4921-910d-f9fff2b347d1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-tljgj\" (UID: \"12bf7b6a-fd62-4921-910d-f9fff2b347d1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367281    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367297    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/21be6874-6138-4a00-9c6e-aa1841eb3230-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.367286    6099 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367314    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c48d07-5599-4a74-94d4-d6488bde2827-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367335    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.367350    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:07.867338919 +0000 UTC m=+1.724246655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367394    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367413    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/070879ac-09e8-4fdb-b55f-e52a9998552f-env-overrides\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367441    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367477    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-run-k8s-cni-cncf-io\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367479    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367507    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c48d07-5599-4a74-94d4-d6488bde2827-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367524    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/070879ac-09e8-4fdb-b55f-e52a9998552f-env-overrides\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367523    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b8c48d07-5599-4a74-94d4-d6488bde2827-metrics-server-audit-profiles\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:35:07.367833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367554    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmnmx\" (UniqueName: \"kubernetes.io/projected/b8c48d07-5599-4a74-94d4-d6488bde2827-kube-api-access-qmnmx\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367878    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-sysctl-conf\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367915    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-sys\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.367949    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/962a4e53-88b6-4866-abc5-108b6d5ce784-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.368010    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.368036    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-hostroot\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.368110    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b8c48d07-5599-4a74-94d4-d6488bde2827-metrics-server-audit-profiles\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.368169    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.368325    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/962a4e53-88b6-4866-abc5-108b6d5ce784-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.368355    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.368471    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-cni-bin\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.368506    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.368517    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.368554    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-etc-kubernetes\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.368591    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b8c48d07-5599-4a74-94d4-d6488bde2827-secret-metrics-server-client-certs\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.368665    6099 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.368669    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32086945-d23d-49bb-a876-fab4cef52ec6-host-slash\") pod \"iptables-alerter-whx68\" (UID: \"32086945-d23d-49bb-a876-fab4cef52ec6\") " pod="openshift-network-operator/iptables-alerter-whx68"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.368722    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5975a832-beb3-4e76-98ad-aef36a2ccce2-host\") pod \"node-ca-d9lhd\" (UID: \"5975a832-beb3-4e76-98ad-aef36a2ccce2\") " pod="openshift-image-registry/node-ca-d9lhd"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.368763    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-secret-telemeter-client\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.368911    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b8c48d07-5599-4a74-94d4-d6488bde2827-secret-metrics-server-client-certs\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.369014    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:07.86898606 +0000 UTC m=+1.725893859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"service-ca" not registered
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.369271    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-federate-client-tls\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.369323    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-secret-telemeter-client\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.369648    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-federate-client-tls\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.369726    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47d34735-e4b1-4073-b6de-98a9a720432b-konnectivity-ca\") pod \"konnectivity-agent-v6rdh\" (UID: \"47d34735-e4b1-4073-b6de-98a9a720432b\") " pod="kube-system/konnectivity-agent-v6rdh"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.369802    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-crio-socket\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.369834    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-registration-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.369874    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7-tmp-dir\") pod \"node-resolver-b2bnk\" (UID: \"b45c13d7-29c0-4054-9b7d-fd33ecec9cf7\") " pod="openshift-dns/node-resolver-b2bnk"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.369905    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-web-config\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.369936    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-var-lib-kubelet\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.369966    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-conf-dir\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370030    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-run-multus-certs\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370070    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-etc-selinux\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370095    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-out\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370123    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-lib-modules\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370151    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d996f136-ba95-4636-a870-94520d3eff0f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370180    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-sys-fs\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370207    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:35:07.370266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370237    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-config\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370366    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/070879ac-09e8-4fdb-b55f-e52a9998552f-ovnkube-script-lib\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370393    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f53ec687-40dd-4358-bc05-83d064d9d082-root\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370425    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d996f136-ba95-4636-a870-94520d3eff0f-cni-binary-copy\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370458    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7lkm\" (UniqueName: \"kubernetes.io/projected/21be6874-6138-4a00-9c6e-aa1841eb3230-kube-api-access-p7lkm\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370490    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370516    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6gmn\" (UniqueName: \"kubernetes.io/projected/f53ec687-40dd-4358-bc05-83d064d9d082-kube-api-access-l6gmn\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370545    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370575    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370605    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370635    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370660    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-run-openvswitch\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370690    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpbnj\" (UniqueName: \"kubernetes.io/projected/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-kube-api-access-fpbnj\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370743    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-socket-dir-parent\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370773    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmqr7\" (UniqueName: \"kubernetes.io/projected/5ac72880-98ab-4093-b7aa-903a37c8113e-kube-api-access-bmqr7\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370803    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.370830    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/962a4e53-88b6-4866-abc5-108b6d5ce784-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371041    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371075    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/070879ac-09e8-4fdb-b55f-e52a9998552f-ovnkube-config\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371106    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47d34735-e4b1-4073-b6de-98a9a720432b-agent-certs\") pod \"konnectivity-agent-v6rdh\" (UID: \"47d34735-e4b1-4073-b6de-98a9a720432b\") " pod="kube-system/konnectivity-agent-v6rdh"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371133    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371165    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f24vm\" (UniqueName: \"kubernetes.io/projected/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-kube-api-access-f24vm\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371196    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371228    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371258    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwm8g\" (UniqueName: \"kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-kube-api-access-fwm8g\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371283    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4tjg\" (UniqueName: \"kubernetes.io/projected/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7-kube-api-access-f4tjg\") pod \"node-resolver-b2bnk\" (UID: \"b45c13d7-29c0-4054-9b7d-fd33ecec9cf7\") " pod="openshift-dns/node-resolver-b2bnk"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371313    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-257nk\" (UniqueName: \"kubernetes.io/projected/070879ac-09e8-4fdb-b55f-e52a9998552f-kube-api-access-257nk\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371349    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b7f00b4a-044e-49ab-8f43-4351590692e8-kubelet-config\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371381    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d996f136-ba95-4636-a870-94520d3eff0f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371407    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371448    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371477    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-grpc-tls\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371508    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b8c48d07-5599-4a74-94d4-d6488bde2827-secret-metrics-server-tls\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371534    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-config-out\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371567    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371599    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ncsp\" (UniqueName: \"kubernetes.io/projected/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-kube-api-access-9ncsp\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371632    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j86q\" (UniqueName: \"kubernetes.io/projected/0b308ab3-9eb7-4be8-964f-ff94af40b883-kube-api-access-6j86q\") pod \"ingress-canary-xfk62\" (UID: \"0b308ab3-9eb7-4be8-964f-ff94af40b883\") " pod="openshift-ingress-canary/ingress-canary-xfk62"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371662    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-sysctl-d\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371689    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/962a4e53-88b6-4866-abc5-108b6d5ce784-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371741    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhmk\" (UniqueName: \"kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk\") pod \"network-check-target-62gpq\" (UID: \"c9803898-12ec-4a07-9a8f-1eef3f89235b\") " pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371785    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/070879ac-09e8-4fdb-b55f-e52a9998552f-ovn-node-metrics-cert\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371815    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-metrics-tls\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371842    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nbn8\" (UniqueName: \"kubernetes.io/projected/3a145ece-ad48-43c2-a259-b9590802cfe8-kube-api-access-8nbn8\") pod \"network-metrics-daemon-wb6st\" (UID: \"3a145ece-ad48-43c2-a259-b9590802cfe8\") " pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371874    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-wtmp\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371907    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371938    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-cni-netd\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.371951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.371970    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.372001    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.372570    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.372602    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-systemd\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.372642    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-tmp\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.372724    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rmw\" (UniqueName: \"kubernetes.io/projected/32086945-d23d-49bb-a876-fab4cef52ec6-kube-api-access-d9rmw\") pod \"iptables-alerter-whx68\" (UID: \"32086945-d23d-49bb-a876-fab4cef52ec6\") " pod="openshift-network-operator/iptables-alerter-whx68"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.372754    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d996f136-ba95-4636-a870-94520d3eff0f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.372766    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnd4w\" (UniqueName: \"kubernetes.io/projected/5975a832-beb3-4e76-98ad-aef36a2ccce2-kube-api-access-cnd4w\") pod \"node-ca-d9lhd\" (UID: \"5975a832-beb3-4e76-98ad-aef36a2ccce2\") " pod="openshift-image-registry/node-ca-d9lhd"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.372799    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-tmp\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.372799    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.372831    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-cni-dir\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.372869    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d996f136-ba95-4636-a870-94520d3eff0f-system-cni-dir\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.372903    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d996f136-ba95-4636-a870-94520d3eff0f-cnibin\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373063    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373081    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373100    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-tmp\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373115    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373253    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373267    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373305    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.372834    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/070879ac-09e8-4fdb-b55f-e52a9998552f-ovnkube-config\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373479    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-tmp\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373504    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-out\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373514    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-config-out\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373507    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47d34735-e4b1-4073-b6de-98a9a720432b-konnectivity-ca\") pod \"konnectivity-agent-v6rdh\" (UID: \"47d34735-e4b1-4073-b6de-98a9a720432b\") " pod="kube-system/konnectivity-agent-v6rdh"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373523    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373545    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b8c48d07-5599-4a74-94d4-d6488bde2827-secret-metrics-server-tls\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373566    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373624    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373684    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5975a832-beb3-4e76-98ad-aef36a2ccce2-serviceca\") pod \"node-ca-d9lhd\" (UID: \"5975a832-beb3-4e76-98ad-aef36a2ccce2\") " pod="openshift-image-registry/node-ca-d9lhd"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373759    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373789    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-tmp-dir\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373827    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.374061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.373947    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/962a4e53-88b6-4866-abc5-108b6d5ce784-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:35:07.375092 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.374340    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-serving-certs-ca-bundle\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:35:07.375092 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.374404    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:35:07.375092 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.374424    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/070879ac-09e8-4fdb-b55f-e52a9998552f-ovn-node-metrics-cert\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.375092 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.374597    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47d34735-e4b1-4073-b6de-98a9a720432b-agent-certs\") pod \"konnectivity-agent-v6rdh\" (UID: \"47d34735-e4b1-4073-b6de-98a9a720432b\") " pod="kube-system/konnectivity-agent-v6rdh"
Oct 30 00:35:07.375092 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.374613    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/962a4e53-88b6-4866-abc5-108b6d5ce784-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:35:07.375092 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.374724    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:07.375092 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.374735    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.375092 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.374766    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:07.375092 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.374842    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-metrics-tls\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:35:07.375092 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.374930    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.375092 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.375014    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-web-config\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.375092 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.375022    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.375560 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.375119    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7-tmp-dir\") pod \"node-resolver-b2bnk\" (UID: \"b45c13d7-29c0-4054-9b7d-fd33ecec9cf7\") " pod="openshift-dns/node-resolver-b2bnk"
Oct 30 00:35:07.375560 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.375128    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.375560 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.375230    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/070879ac-09e8-4fdb-b55f-e52a9998552f-ovnkube-script-lib\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.375560 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.375418    6099 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 00:35:07.375560 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.375419    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-grpc-tls\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:07.375560 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.375432    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d996f136-ba95-4636-a870-94520d3eff0f-cni-binary-copy\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:35:07.375560 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.375481    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.375910 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.375610    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.375910 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.375698    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-telemeter-client-tls\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:35:07.375910 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.375747    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:07.875732171 +0000 UTC m=+1.732639907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 00:35:07.375910 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.375771    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-tls\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.375910 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.375804    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-tmp-dir\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:35:07.375910 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.375804    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-tls\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:07.375910 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.375844    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-var-lib-openvswitch\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.375910 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.375871    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.376279 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.375925    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-cnibin\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.376279 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.375930    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-config\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.376279 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.375958    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d996f136-ba95-4636-a870-94520d3eff0f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:35:07.376279 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376000    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:07.376279 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376025    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-tls\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:07.376279 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376095    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b8c48d07-5599-4a74-94d4-d6488bde2827-client-ca-bundle\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:35:07.376279 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376142    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5kb7\" (UniqueName: \"kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-kube-api-access-j5kb7\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.376279 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376143    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d996f136-ba95-4636-a870-94520d3eff0f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:35:07.376279 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376183    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-tls\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.376279 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376240    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5975a832-beb3-4e76-98ad-aef36a2ccce2-serviceca\") pod \"node-ca-d9lhd\" (UID: \"5975a832-beb3-4e76-98ad-aef36a2ccce2\") " pod="openshift-image-registry/node-ca-d9lhd"
Oct 30 00:35:07.376725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376283    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-node-log\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.376725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376316    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c5282344-9beb-485c-9525-6b09bdda0804-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:07.376725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376389    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.376725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376426    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ac72880-98ab-4093-b7aa-903a37c8113e-cni-binary-copy\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.376725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376461    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b8c48d07-5599-4a74-94d4-d6488bde2827-client-ca-bundle\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:35:07.376725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376475    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc8e4d97-4c7f-450d-961c-528c992a29b2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:35:07.376725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376514    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-var-lib-kubelet\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.376725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376529    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 00:35:07.376725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376571    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 00:35:07.376725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376656    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ac72880-98ab-4093-b7aa-903a37c8113e-cni-binary-copy\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.376725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376676    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-tuned\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.377198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376740    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.377198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376778    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:07.377198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376804    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-tuned\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.377198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376822    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-textfile\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.377198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376866    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc8e4d97-4c7f-450d-961c-528c992a29b2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:35:07.377198 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.376870    6099 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 00:35:07.377198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376940    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-textfile\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.377198 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.376958    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:07.876942634 +0000 UTC m=+1.733850381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"console-config" not registered
Oct 30 00:35:07.377198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.376988    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-var-lib-cni-bin\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.377198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.377029    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-daemon-config\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.377198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.377032    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.377198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.377070    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 00:35:07.377198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.377071    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5282344-9beb-485c-9525-6b09bdda0804-metrics-client-ca\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:07.377198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.377116    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 00:35:07.377683 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.377207    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-daemon-config\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.377683 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.377256    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-slash\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.377683 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.377298    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-run-netns\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.377683 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.377320    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5282344-9beb-485c-9525-6b09bdda0804-metrics-client-ca\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:07.377683 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.377673    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.378131 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.378115    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.383126 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.383110    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzmdn\" (UniqueName: \"kubernetes.io/projected/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-kube-api-access-xzmdn\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:35:07.384017 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.384000    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.384622 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.384607    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7spz\" (UniqueName: \"kubernetes.io/projected/12bf7b6a-fd62-4921-910d-f9fff2b347d1-kube-api-access-j7spz\") pod \"monitoring-plugin-77f4f76594-tljgj\" (UID: \"12bf7b6a-fd62-4921-910d-f9fff2b347d1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj"
Oct 30 00:35:07.384787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.384772    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhjkw\" (UniqueName: \"kubernetes.io/projected/962a4e53-88b6-4866-abc5-108b6d5ce784-kube-api-access-lhjkw\") pod \"openshift-state-metrics-7bd7489957-4qfxh\" (UID: \"962a4e53-88b6-4866-abc5-108b6d5ce784\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qfxh"
Oct 30 00:35:07.384825 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.384800    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkp59\" (UniqueName: \"kubernetes.io/projected/bc8e4d97-4c7f-450d-961c-528c992a29b2-kube-api-access-mkp59\") pod \"kube-state-metrics-5dc8d7d89b-pcmnz\" (UID: \"bc8e4d97-4c7f-450d-961c-528c992a29b2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pcmnz"
Oct 30 00:35:07.385178 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.385162    6099 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 00:35:07.385246 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.385181    6099 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 00:35:07.385246 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.385194    6099 projected.go:196] Error preparing data for projected volume kube-api-access-r59hp for pod openshift-console/console-57449f4d45-vtkst: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 00:35:07.385338 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.385272    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:07.885257009 +0000 UTC m=+1.742164764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r59hp" (UniqueName: "kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 00:35:07.385583 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.385565    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpw5w\" (UniqueName: \"kubernetes.io/projected/c5282344-9beb-485c-9525-6b09bdda0804-kube-api-access-wpw5w\") pod \"thanos-querier-676949fd44-65zsj\" (UID: \"c5282344-9beb-485c-9525-6b09bdda0804\") " pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:07.385614 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.385602    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhvbj\" (UniqueName: \"kubernetes.io/projected/733527bc-83b7-423c-a79e-1de279f5a607-kube-api-access-jhvbj\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.386746 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.386728    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmnmx\" (UniqueName: \"kubernetes.io/projected/b8c48d07-5599-4a74-94d4-d6488bde2827-kube-api-access-qmnmx\") pod \"metrics-server-7c55cf7f78-h65h8\" (UID: \"b8c48d07-5599-4a74-94d4-d6488bde2827\") " pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:35:07.386979 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.386959    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:35:07.391366 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.391346    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f24vm\" (UniqueName: \"kubernetes.io/projected/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-kube-api-access-f24vm\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.391515 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.391498    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4tjg\" (UniqueName: \"kubernetes.io/projected/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7-kube-api-access-f4tjg\") pod \"node-resolver-b2bnk\" (UID: \"b45c13d7-29c0-4054-9b7d-fd33ecec9cf7\") " pod="openshift-dns/node-resolver-b2bnk"
Oct 30 00:35:07.391581 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.391553    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nbn8\" (UniqueName: \"kubernetes.io/projected/3a145ece-ad48-43c2-a259-b9590802cfe8-kube-api-access-8nbn8\") pod \"network-metrics-daemon-wb6st\" (UID: \"3a145ece-ad48-43c2-a259-b9590802cfe8\") " pod="openshift-multus/network-metrics-daemon-wb6st"
Oct 30 00:35:07.391734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.391694    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbtjm\" (UniqueName: \"kubernetes.io/projected/d996f136-ba95-4636-a870-94520d3eff0f-kube-api-access-vbtjm\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:35:07.392762 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.392739    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ncsp\" (UniqueName: \"kubernetes.io/projected/ddb2dce5-50be-42dc-88f4-ecb4d6a29d28-kube-api-access-9ncsp\") pod \"telemeter-client-6ddf586968-ml4vv\" (UID: \"ddb2dce5-50be-42dc-88f4-ecb4d6a29d28\") " pod="openshift-monitoring/telemeter-client-6ddf586968-ml4vv"
Oct 30 00:35:07.392950 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.392931    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmqr7\" (UniqueName: \"kubernetes.io/projected/5ac72880-98ab-4093-b7aa-903a37c8113e-kube-api-access-bmqr7\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.393031 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.392930    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbhmk\" (UniqueName: \"kubernetes.io/projected/c9803898-12ec-4a07-9a8f-1eef3f89235b-kube-api-access-qbhmk\") pod \"network-check-target-62gpq\" (UID: \"c9803898-12ec-4a07-9a8f-1eef3f89235b\") " pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:35:07.393285 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.393270    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpbnj\" (UniqueName: \"kubernetes.io/projected/70c3a600-551a-4ff4-8fbe-e9a2aa1172e3-kube-api-access-fpbnj\") pod \"dns-default-rwwwf\" (UID: \"70c3a600-551a-4ff4-8fbe-e9a2aa1172e3\") " pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:35:07.393379 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.393360    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwm8g\" (UniqueName: \"kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-kube-api-access-fwm8g\") pod \"alertmanager-main-0\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:07.393421 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.393360    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j86q\" (UniqueName: \"kubernetes.io/projected/0b308ab3-9eb7-4be8-964f-ff94af40b883-kube-api-access-6j86q\") pod \"ingress-canary-xfk62\" (UID: \"0b308ab3-9eb7-4be8-964f-ff94af40b883\") " pod="openshift-ingress-canary/ingress-canary-xfk62"
Oct 30 00:35:07.394352 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.394337    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7lkm\" (UniqueName: \"kubernetes.io/projected/21be6874-6138-4a00-9c6e-aa1841eb3230-kube-api-access-p7lkm\") pod \"prometheus-operator-9cd89b74c-ls4dj\" (UID: \"21be6874-6138-4a00-9c6e-aa1841eb3230\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ls4dj"
Oct 30 00:35:07.394739 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.394720    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6gmn\" (UniqueName: \"kubernetes.io/projected/f53ec687-40dd-4358-bc05-83d064d9d082-kube-api-access-l6gmn\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.394784 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.394771    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9rmw\" (UniqueName: \"kubernetes.io/projected/32086945-d23d-49bb-a876-fab4cef52ec6-kube-api-access-d9rmw\") pod \"iptables-alerter-whx68\" (UID: \"32086945-d23d-49bb-a876-fab4cef52ec6\") " pod="openshift-network-operator/iptables-alerter-whx68"
Oct 30 00:35:07.395872 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.395858    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnd4w\" (UniqueName: \"kubernetes.io/projected/5975a832-beb3-4e76-98ad-aef36a2ccce2-kube-api-access-cnd4w\") pod \"node-ca-d9lhd\" (UID: \"5975a832-beb3-4e76-98ad-aef36a2ccce2\") " pod="openshift-image-registry/node-ca-d9lhd"
Oct 30 00:35:07.397117 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.397101    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5kb7\" (UniqueName: \"kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-kube-api-access-j5kb7\") pod \"prometheus-k8s-0\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:07.399389 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.399375    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-257nk\" (UniqueName: \"kubernetes.io/projected/070879ac-09e8-4fdb-b55f-e52a9998552f-kube-api-access-257nk\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.478255 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478195    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-kubernetes\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.478255 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478244    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-system-cni-dir\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.478391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478271    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-device-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.478391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478271    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-kubernetes\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.478391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478296    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-run-ovn\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.478391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478338    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-device-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.478391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478348    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-sysconfig\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.478391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478358    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-system-cni-dir\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.478391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478381    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-run-ovn\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.478391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478374    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-kubelet\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478408    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-sysconfig\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478412    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b7f00b4a-044e-49ab-8f43-4351590692e8-dbus\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478438    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-socket-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478467    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f53ec687-40dd-4358-bc05-83d064d9d082-sys\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478466    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-kubelet\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478508    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-socket-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478509    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f53ec687-40dd-4358-bc05-83d064d9d082-sys\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478511    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b7f00b4a-044e-49ab-8f43-4351590692e8-dbus\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478530    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-host\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478567    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-var-lib-cni-multus\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478592    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7-hosts-file\") pod \"node-resolver-b2bnk\" (UID: \"b45c13d7-29c0-4054-9b7d-fd33ecec9cf7\") " pod="openshift-dns/node-resolver-b2bnk"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478615    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-host\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478617    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-etc-openvswitch\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478649    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-etc-openvswitch\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478661    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-log-socket\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478670    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-var-lib-cni-multus\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478680    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b45c13d7-29c0-4054-9b7d-fd33ecec9cf7-hosts-file\") pod \"node-resolver-b2bnk\" (UID: \"b45c13d7-29c0-4054-9b7d-fd33ecec9cf7\") " pod="openshift-dns/node-resolver-b2bnk"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478692    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-log-socket\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478729    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-run-netns\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478747    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-run-netns\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.478758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478759    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-modprobe-d\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478783    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-run-systemd\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478806    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-run\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478839    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-systemd-units\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478850    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-run-systemd\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478854    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-run\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478884    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-run-k8s-cni-cncf-io\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478902    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-sysctl-conf\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478900    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-systemd-units\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478924    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-sys\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478935    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-run-k8s-cni-cncf-io\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478947    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-hostroot\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478949    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-modprobe-d\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478967    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-cni-bin\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478993    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-sys\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.478999    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-etc-kubernetes\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479022    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-hostroot\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479024    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32086945-d23d-49bb-a876-fab4cef52ec6-host-slash\") pod \"iptables-alerter-whx68\" (UID: \"32086945-d23d-49bb-a876-fab4cef52ec6\") " pod="openshift-network-operator/iptables-alerter-whx68"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479032    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-cni-bin\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479037    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-sysctl-conf\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479058    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-etc-kubernetes\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479060    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5975a832-beb3-4e76-98ad-aef36a2ccce2-host\") pod \"node-ca-d9lhd\" (UID: \"5975a832-beb3-4e76-98ad-aef36a2ccce2\") " pod="openshift-image-registry/node-ca-d9lhd"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479097    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-crio-socket\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479072    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32086945-d23d-49bb-a876-fab4cef52ec6-host-slash\") pod \"iptables-alerter-whx68\" (UID: \"32086945-d23d-49bb-a876-fab4cef52ec6\") " pod="openshift-network-operator/iptables-alerter-whx68"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479122    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-registration-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479126    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5975a832-beb3-4e76-98ad-aef36a2ccce2-host\") pod \"node-ca-d9lhd\" (UID: \"5975a832-beb3-4e76-98ad-aef36a2ccce2\") " pod="openshift-image-registry/node-ca-d9lhd"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479154    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-var-lib-kubelet\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479171    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-conf-dir\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479175    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-registration-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479168    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6-crio-socket\") pod \"insights-runtime-extractor-wvmfk\" (UID: \"2991faa6-6a5e-4fbb-8b0b-f17daa3d38a6\") " pod="openshift-insights/insights-runtime-extractor-wvmfk"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479190    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-run-multus-certs\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479212    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-etc-selinux\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479215    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-var-lib-kubelet\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479224    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-conf-dir\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479227    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-lib-modules\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479255    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-run-multus-certs\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479263    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-etc-selinux\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479265    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d996f136-ba95-4636-a870-94520d3eff0f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479290    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-sys-fs\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479314    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f53ec687-40dd-4358-bc05-83d064d9d082-root\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479322    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d996f136-ba95-4636-a870-94520d3eff0f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479330    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-lib-modules\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479341    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479369    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f53ec687-40dd-4358-bc05-83d064d9d082-root\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479377    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-sys-fs\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479401    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-run-openvswitch\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479410    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479431    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-socket-dir-parent\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479467    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b7f00b4a-044e-49ab-8f43-4351590692e8-kubelet-config\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479480    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-run-openvswitch\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479513    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-sysctl-d\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479522    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b7f00b4a-044e-49ab-8f43-4351590692e8-kubelet-config\") pod \"global-pull-secret-syncer-7vsmg\" (UID: \"b7f00b4a-044e-49ab-8f43-4351590692e8\") " pod="kube-system/global-pull-secret-syncer-7vsmg"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479515    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-socket-dir-parent\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479543    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-wtmp\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479567    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-cni-netd\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479582    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f53ec687-40dd-4358-bc05-83d064d9d082-node-exporter-wtmp\") pod \"node-exporter-wb627\" (UID: \"f53ec687-40dd-4358-bc05-83d064d9d082\") " pod="openshift-monitoring/node-exporter-wb627"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479592    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-systemd\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479611    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-sysctl-d\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479620    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-cni-dir\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479645    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d996f136-ba95-4636-a870-94520d3eff0f-system-cni-dir\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479619    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-cni-netd\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479664    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-etc-systemd\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.479630 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479666    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d996f136-ba95-4636-a870-94520d3eff0f-cnibin\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479692    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d996f136-ba95-4636-a870-94520d3eff0f-system-cni-dir\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479697    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d996f136-ba95-4636-a870-94520d3eff0f-cnibin\") pod \"multus-additional-cni-plugins-gmgzr\" (UID: \"d996f136-ba95-4636-a870-94520d3eff0f\") " pod="openshift-multus/multus-additional-cni-plugins-gmgzr"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479695    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-multus-cni-dir\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479731    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479771    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479773    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-var-lib-openvswitch\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479798    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-var-lib-openvswitch\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479807    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479834    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-cnibin\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479849    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/733527bc-83b7-423c-a79e-1de279f5a607-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nk25v\" (UID: \"733527bc-83b7-423c-a79e-1de279f5a607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nk25v"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479872    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-node-log\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479900    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-var-lib-kubelet\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479903    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-cnibin\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479938    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-node-log\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479939    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-var-lib-cni-bin\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479969    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ac72880-98ab-4093-b7aa-903a37c8113e-host-var-lib-cni-bin\") pod \"multus-42gzj\" (UID: \"5ac72880-98ab-4093-b7aa-903a37c8113e\") " pod="openshift-multus/multus-42gzj"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.479979    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-slash\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.480010    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b73f4c46-cb05-492e-8dbc-953f3c70b1d5-var-lib-kubelet\") pod \"tuned-9h5hq\" (UID: \"b73f4c46-cb05-492e-8dbc-953f3c70b1d5\") " pod="openshift-cluster-node-tuning-operator/tuned-9h5hq"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.480012    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-slash\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.480012    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-run-netns\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.481693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.480043    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/070879ac-09e8-4fdb-b55f-e52a9998552f-host-run-netns\") pod \"ovnkube-node-vcqkc\" (UID: \"070879ac-09e8-4fdb-b55f-e52a9998552f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:07.596749 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.596721    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rwwwf"
Oct 30 00:35:07.884617 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.884574    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:07.884800 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.884641    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:07.884800 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.884655    6099 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 00:35:07.884800 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.884724    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:07.884800 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.884751    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:08.884731454 +0000 UTC m=+2.741639208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 00:35:07.884800 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.884770    6099 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 00:35:07.884800 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.884787    6099 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 00:35:07.884998 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.884817    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:07.884998 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.884861    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:08.884848572 +0000 UTC m=+2.741756308 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"service-ca" not registered
Oct 30 00:35:07.884998 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.884872    6099 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 00:35:07.884998 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.884885    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:07.884998 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.884925    6099 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 00:35:07.884998 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.884940    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:08.884924755 +0000 UTC m=+2.741832520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"console-config" not registered
Oct 30 00:35:07.884998 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.884959    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:08.884949523 +0000 UTC m=+2.741857255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 00:35:07.884998 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.884985    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:08.88496599 +0000 UTC m=+2.741873723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 00:35:07.985866 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:07.985835    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59hp\" (UniqueName: \"kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:07.986003 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.985977    6099 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 00:35:07.986003 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.985998    6099 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 00:35:07.986090 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.986008    6099 projected.go:196] Error preparing data for projected volume kube-api-access-r59hp for pod openshift-console/console-57449f4d45-vtkst: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 00:35:07.986090 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:07.986062    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:08.986049817 +0000 UTC m=+2.842957589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r59hp" (UniqueName: "kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 00:35:08.895872 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:08.895835    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:08.896302 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:08.895895    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:08.896302 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:08.895925    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:08.896302 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:08.895973    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:08.896302 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:08.896003    6099 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 00:35:08.896302 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:08.896023    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:08.896302 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:08.896043    6099 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 00:35:08.896302 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:08.896056    6099 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 00:35:08.896302 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:08.896087    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:10.896066021 +0000 UTC m=+4.752973760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 00:35:08.896302 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:08.896091    6099 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 00:35:08.896302 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:08.896109    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:10.896098623 +0000 UTC m=+4.753006359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"service-ca" not registered
Oct 30 00:35:08.896302 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:08.896125    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:10.896115624 +0000 UTC m=+4.753023359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 00:35:08.896302 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:08.896139    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:10.896132053 +0000 UTC m=+4.753039787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"console-config" not registered
Oct 30 00:35:08.896302 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:08.896136    6099 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 00:35:08.896302 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:08.896227    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:10.896214302 +0000 UTC m=+4.753122041 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 00:35:08.996954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:08.996923    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59hp\" (UniqueName: \"kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:08.997137 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:08.997114    6099 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 00:35:08.997195 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:08.997145    6099 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 00:35:08.997195 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:08.997159    6099 projected.go:196] Error preparing data for projected volume kube-api-access-r59hp for pod openshift-console/console-57449f4d45-vtkst: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 00:35:08.997253 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:08.997221    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:10.997204177 +0000 UTC m=+4.854111943 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r59hp" (UniqueName: "kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 00:35:09.005227 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:09.004243    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-676949fd44-65zsj"
Oct 30 00:35:09.475998 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:09.475964    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:09.971335 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:09.971299    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:10.274086 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:10.274005    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:35:10.344401 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:10.344370    6099 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:35:10.915868 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:10.915819    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:10.916147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:10.915885    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:10.916147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:10.915916    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:10.916147 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:10.915945    6099 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 00:35:10.916147 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:10.916012    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:14.915997911 +0000 UTC m=+8.772905644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 00:35:10.916147 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:10.916011    6099 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 00:35:10.916147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:10.915953    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:10.916147 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:10.916015    6099 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 00:35:10.916147 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:10.916069    6099 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 00:35:10.916147 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:10.916079    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:14.916063135 +0000 UTC m=+8.772970889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"console-config" not registered
Oct 30 00:35:10.916147 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:10.916127    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:14.91611821 +0000 UTC m=+8.773025947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 00:35:10.916147 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:10.916139    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:14.916133764 +0000 UTC m=+8.773041497 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 00:35:10.916508 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:10.916184    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:10.916508 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:10.916252    6099 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 00:35:10.916508 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:10.916285    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:14.916274333 +0000 UTC m=+8.773182066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"service-ca" not registered
Oct 30 00:35:11.017525 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:11.017497    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59hp\" (UniqueName: \"kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:11.017900 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:11.017637    6099 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 00:35:11.017900 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:11.017654    6099 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 00:35:11.017900 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:11.017665    6099 projected.go:196] Error preparing data for projected volume kube-api-access-r59hp for pod openshift-console/console-57449f4d45-vtkst: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 00:35:11.017900 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:11.017724    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:15.01769492 +0000 UTC m=+8.874602688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r59hp" (UniqueName: "kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 00:35:11.756199 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:11.756172    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:14.958645 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:14.958609    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:14.959075 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:14.958654    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:14.959075 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:14.958731    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:14.959075 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:14.958740    6099 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 00:35:14.959075 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:14.958765    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:14.959075 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:14.958832    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:22.958811625 +0000 UTC m=+16.815719380 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"console-config" not registered
Oct 30 00:35:14.959075 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:14.958855    6099 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 00:35:14.959075 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:14.958858    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:14.959075 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:14.958868    6099 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 00:35:14.959075 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:14.958912    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:22.95890064 +0000 UTC m=+16.815808372 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"service-ca" not registered
Oct 30 00:35:14.959075 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:14.958856    6099 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 00:35:14.959075 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:14.958928    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:22.958919427 +0000 UTC m=+16.815827161 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 00:35:14.959075 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:14.958950    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:22.958937063 +0000 UTC m=+16.815844805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 00:35:14.959075 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:14.958952    6099 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 00:35:14.959075 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:14.958983    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:22.958976396 +0000 UTC m=+16.815884160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 00:35:15.059625 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:15.059591    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59hp\" (UniqueName: \"kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:15.059825 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:15.059802    6099 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 00:35:15.059903 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:15.059832    6099 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 00:35:15.059903 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:15.059847    6099 projected.go:196] Error preparing data for projected volume kube-api-access-r59hp for pod openshift-console/console-57449f4d45-vtkst: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 00:35:15.060084 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:15.059925    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:23.05990574 +0000 UTC m=+16.916813476 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r59hp" (UniqueName: "kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 00:35:16.628548 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:16.628513    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:35:16.972173 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:16.972104    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj"
Oct 30 00:35:21.006936 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:21.006854    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcqkc"
Oct 30 00:35:23.033170 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:23.033140    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:23.033569 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:23.033198    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:23.033569 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:23.033243    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:23.033569 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:23.033251    6099 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 00:35:23.033569 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:23.033317    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:39.033299752 +0000 UTC m=+32.890207485 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 00:35:23.033569 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:23.033320    6099 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 00:35:23.033569 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:23.033339    6099 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 00:35:23.033569 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:23.033376    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:39.033366192 +0000 UTC m=+32.890273948 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 00:35:23.033569 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:23.033411    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:23.033569 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:23.033421    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:39.033407401 +0000 UTC m=+32.890315134 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"console-config" not registered
Oct 30 00:35:23.033569 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:23.033467    6099 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 00:35:23.033569 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:23.033493    6099 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 00:35:23.033569 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:23.033500    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:39.03349072 +0000 UTC m=+32.890398490 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 00:35:23.033569 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:23.033465    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:23.033569 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:23.033517    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:39.03351071 +0000 UTC m=+32.890418492 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : object "openshift-console"/"service-ca" not registered
Oct 30 00:35:23.135032 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:23.135002    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59hp\" (UniqueName: \"kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp\") pod \"console-57449f4d45-vtkst\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") " pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:23.135179 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:23.135117    6099 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 00:35:23.135179 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:23.135132    6099 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 00:35:23.135179 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:23.135141    6099 projected.go:196] Error preparing data for projected volume kube-api-access-r59hp for pod openshift-console/console-57449f4d45-vtkst: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 00:35:23.135274 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:23.135186    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp podName:fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd nodeName:}" failed. No retries permitted until 2026-10-30 00:35:39.135174349 +0000 UTC m=+32.992082081 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r59hp" (UniqueName: "kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp") pod "console-57449f4d45-vtkst" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 00:35:30.202851 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:30.202815    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7c55cf7f78-h65h8"
Oct 30 00:35:32.282806 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.282744    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-57449f4d45-vtkst" podUID="fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" containerName="console" containerID="cri-o://267053f4f6c5e5cb54f2f2febf56a0d4e71d0fde1a936ac35691f8033722299e" gracePeriod=15
Oct 30 00:35:32.430014 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.429979    6099 generic.go:356] "Generic (PLEG): container finished" podID="fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" containerID="267053f4f6c5e5cb54f2f2febf56a0d4e71d0fde1a936ac35691f8033722299e" exitCode=0
Oct 30 00:35:32.430148 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.430044    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57449f4d45-vtkst" event={"ID":"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd","Type":"ContainerDied","Data":"267053f4f6c5e5cb54f2f2febf56a0d4e71d0fde1a936ac35691f8033722299e"}
Oct 30 00:35:32.520955 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.520932    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:32.624591 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.624561    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config\") pod \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") "
Oct 30 00:35:32.624779 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.624600    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-tmp\") pod \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") "
Oct 30 00:35:32.624779 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.624629    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert\") pod \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") "
Oct 30 00:35:32.624917 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.624769    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert\") pod \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") "
Oct 30 00:35:32.624917 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.624871    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp\" (UniqueName: \"kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp\") pod \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") "
Oct 30 00:35:32.625032 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.624910    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-tmp" pod "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:35:32.625032 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.624916    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config\") pod \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") "
Oct 30 00:35:32.625151 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.625029    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca\") pod \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\" (UID: \"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd\") "
Oct 30 00:35:32.625151 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.625061    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert" pod "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:35:32.625428 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.625412    6099 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-tmp\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:32.625495 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.625434    6099 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-oauth-serving-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:32.625717 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.625676    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config" pod "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:35:32.625909 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.625880    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca" pod "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:35:32.627167 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.627148    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config" pod "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:32.627867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.627827    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert" pod "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:32.627944 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.627903    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp" pod "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" (UID: "fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd"). InnerVolumeSpecName "kube-api-access-r59hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:35:32.726091 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.726064    6099 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-oauth-config\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:32.726091 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.726086    6099 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-serving-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:32.726091 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.726096    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r59hp\" (UniqueName: \"kubernetes.io/projected/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-kube-api-access-r59hp\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:32.726265 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.726106    6099 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-console-config\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:32.726265 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:32.726115    6099 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:33.438600 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:33.438567    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57449f4d45-vtkst" event={"ID":"fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd","Type":"ContainerDied","Data":"4fa4df476257e406e2a2ef2135a8a7a2d2f203d6bbd313427afb78fe163973a0"}
Oct 30 00:35:33.439038 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:33.438612    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57449f4d45-vtkst"
Oct 30 00:35:33.439038 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:33.438645    6099 scope.go:122] "RemoveContainer" containerID="267053f4f6c5e5cb54f2f2febf56a0d4e71d0fde1a936ac35691f8033722299e"
Oct 30 00:35:33.482435 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:33.482411    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57449f4d45-vtkst"]
Oct 30 00:35:33.492302 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:33.492282    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57449f4d45-vtkst"]
Oct 30 00:35:34.258745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:34.258693    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" path="/var/lib/kubelet/pods/fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd/volumes"
Oct 30 00:35:35.662295 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:35.662264    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 00:35:35.662871 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:35.662565    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="alertmanager" containerID="cri-o://273b61e0df7f8e20c3a0d13e72a028b7f29b04f04826d53026d3244a3444304f" gracePeriod=120
Oct 30 00:35:35.662871 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:35.662606    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="config-reloader" containerID="cri-o://6a69994adc0ccc20c20c18cc3c4b2db5e34a2175084c002010effc404a0b33b6" gracePeriod=120
Oct 30 00:35:35.662871 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:35.662655    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="kube-rbac-proxy" containerID="cri-o://5515e80e8e28805e47ca2d5dd4127719c0b4fd426c264e0ec0becc2f84687f9d" gracePeriod=120
Oct 30 00:35:35.662871 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:35.662629    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="kube-rbac-proxy-web" containerID="cri-o://bf396d37b29bfa2006e3935fa69145a8acb684784ba731ede764600f0f477060" gracePeriod=120
Oct 30 00:35:35.662871 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:35.662831    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="kube-rbac-proxy-metric" containerID="cri-o://56e4c08f9e42d3ea818b5e4c8fd1bcf4ed93ca026c3e47fb577c6e0a6e801432" gracePeriod=120
Oct 30 00:35:35.662871 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:35.662846    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="prom-label-proxy" containerID="cri-o://23deb9e025e27d9e7dbb292cd389ecc5fcb9c5371ebe72e02ecfabdcbde56444" gracePeriod=120
Oct 30 00:35:36.453585 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:36.453550    6099 generic.go:356] "Generic (PLEG): container finished" podID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerID="23deb9e025e27d9e7dbb292cd389ecc5fcb9c5371ebe72e02ecfabdcbde56444" exitCode=0
Oct 30 00:35:36.453585 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:36.453575    6099 generic.go:356] "Generic (PLEG): container finished" podID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerID="5515e80e8e28805e47ca2d5dd4127719c0b4fd426c264e0ec0becc2f84687f9d" exitCode=0
Oct 30 00:35:36.453585 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:36.453585    6099 generic.go:356] "Generic (PLEG): container finished" podID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerID="6a69994adc0ccc20c20c18cc3c4b2db5e34a2175084c002010effc404a0b33b6" exitCode=0
Oct 30 00:35:36.453585 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:36.453592    6099 generic.go:356] "Generic (PLEG): container finished" podID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerID="273b61e0df7f8e20c3a0d13e72a028b7f29b04f04826d53026d3244a3444304f" exitCode=0
Oct 30 00:35:36.453859 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:36.453623    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"426a04b3-dc76-4b89-8d28-3038652fd7bf","Type":"ContainerDied","Data":"23deb9e025e27d9e7dbb292cd389ecc5fcb9c5371ebe72e02ecfabdcbde56444"}
Oct 30 00:35:36.453859 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:36.453686    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"426a04b3-dc76-4b89-8d28-3038652fd7bf","Type":"ContainerDied","Data":"5515e80e8e28805e47ca2d5dd4127719c0b4fd426c264e0ec0becc2f84687f9d"}
Oct 30 00:35:36.453859 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:36.453731    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"426a04b3-dc76-4b89-8d28-3038652fd7bf","Type":"ContainerDied","Data":"6a69994adc0ccc20c20c18cc3c4b2db5e34a2175084c002010effc404a0b33b6"}
Oct 30 00:35:36.453859 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:36.453757    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"426a04b3-dc76-4b89-8d28-3038652fd7bf","Type":"ContainerDied","Data":"273b61e0df7f8e20c3a0d13e72a028b7f29b04f04826d53026d3244a3444304f"}
Oct 30 00:35:36.958417 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:36.958387    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.069747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.066943    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-volume\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-volume\") pod \"426a04b3-dc76-4b89-8d28-3038652fd7bf\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") "
Oct 30 00:35:37.069747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.067002    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"426a04b3-dc76-4b89-8d28-3038652fd7bf\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") "
Oct 30 00:35:37.069747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.067040    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-metrics-client-ca\") pod \"426a04b3-dc76-4b89-8d28-3038652fd7bf\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") "
Oct 30 00:35:37.069747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.067069    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-kube-api-access-fwm8g\" (UniqueName: \"kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-kube-api-access-fwm8g\") pod \"426a04b3-dc76-4b89-8d28-3038652fd7bf\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") "
Oct 30 00:35:37.069747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.067109    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-out\" (UniqueName: \"kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-out\") pod \"426a04b3-dc76-4b89-8d28-3038652fd7bf\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") "
Oct 30 00:35:37.069747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.067140    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-trusted-ca-bundle\") pod \"426a04b3-dc76-4b89-8d28-3038652fd7bf\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") "
Oct 30 00:35:37.069747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.067185    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-main-tls\") pod \"426a04b3-dc76-4b89-8d28-3038652fd7bf\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") "
Oct 30 00:35:37.069747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.067212    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-tls-assets\" (UniqueName: \"kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-tls-assets\") pod \"426a04b3-dc76-4b89-8d28-3038652fd7bf\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") "
Oct 30 00:35:37.069747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.067240    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"426a04b3-dc76-4b89-8d28-3038652fd7bf\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") "
Oct 30 00:35:37.069747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.067269    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-cluster-tls-config\") pod \"426a04b3-dc76-4b89-8d28-3038652fd7bf\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") "
Oct 30 00:35:37.069747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.067326    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-main-db\") pod \"426a04b3-dc76-4b89-8d28-3038652fd7bf\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") "
Oct 30 00:35:37.069747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.067370    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-web-config\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-web-config\") pod \"426a04b3-dc76-4b89-8d28-3038652fd7bf\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") "
Oct 30 00:35:37.069747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.067414    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy\") pod \"426a04b3-dc76-4b89-8d28-3038652fd7bf\" (UID: \"426a04b3-dc76-4b89-8d28-3038652fd7bf\") "
Oct 30 00:35:37.078894 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.074956    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-trusted-ca-bundle" pod "426a04b3-dc76-4b89-8d28-3038652fd7bf" (UID: "426a04b3-dc76-4b89-8d28-3038652fd7bf"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:35:37.078894 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.076841    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-metrics-client-ca" pod "426a04b3-dc76-4b89-8d28-3038652fd7bf" (UID: "426a04b3-dc76-4b89-8d28-3038652fd7bf"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:35:37.082527 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.082493    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-main-db" pod "426a04b3-dc76-4b89-8d28-3038652fd7bf" (UID: "426a04b3-dc76-4b89-8d28-3038652fd7bf"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:35:37.082665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.082522    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-main-tls" pod "426a04b3-dc76-4b89-8d28-3038652fd7bf" (UID: "426a04b3-dc76-4b89-8d28-3038652fd7bf"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:37.082782 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.082672    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-kube-api-access-fwm8g" pod "426a04b3-dc76-4b89-8d28-3038652fd7bf" (UID: "426a04b3-dc76-4b89-8d28-3038652fd7bf"). InnerVolumeSpecName "kube-api-access-fwm8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:35:37.089206 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.088995    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-out" pod "426a04b3-dc76-4b89-8d28-3038652fd7bf" (UID: "426a04b3-dc76-4b89-8d28-3038652fd7bf"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:35:37.089206 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.089100    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-web" pod "426a04b3-dc76-4b89-8d28-3038652fd7bf" (UID: "426a04b3-dc76-4b89-8d28-3038652fd7bf"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:37.089206 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.089034    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-metric" pod "426a04b3-dc76-4b89-8d28-3038652fd7bf" (UID: "426a04b3-dc76-4b89-8d28-3038652fd7bf"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:37.089206 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.089166    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-volume" pod "426a04b3-dc76-4b89-8d28-3038652fd7bf" (UID: "426a04b3-dc76-4b89-8d28-3038652fd7bf"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:37.089564 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.089238    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy" pod "426a04b3-dc76-4b89-8d28-3038652fd7bf" (UID: "426a04b3-dc76-4b89-8d28-3038652fd7bf"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:37.090430 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.090402    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-cluster-tls-config" pod "426a04b3-dc76-4b89-8d28-3038652fd7bf" (UID: "426a04b3-dc76-4b89-8d28-3038652fd7bf"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:37.091062 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.091035    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-tls-assets" pod "426a04b3-dc76-4b89-8d28-3038652fd7bf" (UID: "426a04b3-dc76-4b89-8d28-3038652fd7bf"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:35:37.097413 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.097390    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-web-config" pod "426a04b3-dc76-4b89-8d28-3038652fd7bf" (UID: "426a04b3-dc76-4b89-8d28-3038652fd7bf"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:37.168637 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.168603    6099 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:37.168637 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.168632    6099 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:37.168637 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.168644    6099 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:37.168867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.168654    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-metrics-client-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:37.168867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.168665    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwm8g\" (UniqueName: \"kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-kube-api-access-fwm8g\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:37.168867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.168681    6099 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-config-out\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:37.168867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.168690    6099 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:37.168867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.168699    6099 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-main-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:37.168867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.168730    6099 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/426a04b3-dc76-4b89-8d28-3038652fd7bf-tls-assets\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:37.168867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.168739    6099 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:37.168867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.168748    6099 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-cluster-tls-config\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:37.168867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.168756    6099 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/426a04b3-dc76-4b89-8d28-3038652fd7bf-alertmanager-main-db\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:37.168867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.168763    6099 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/426a04b3-dc76-4b89-8d28-3038652fd7bf-web-config\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:37.361821 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.361759    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v6rdh"
Oct 30 00:35:37.460040 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.460003    6099 generic.go:356] "Generic (PLEG): container finished" podID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerID="56e4c08f9e42d3ea818b5e4c8fd1bcf4ed93ca026c3e47fb577c6e0a6e801432" exitCode=0
Oct 30 00:35:37.460040 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.460034    6099 generic.go:356] "Generic (PLEG): container finished" podID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerID="bf396d37b29bfa2006e3935fa69145a8acb684784ba731ede764600f0f477060" exitCode=0
Oct 30 00:35:37.460040 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.460032    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"426a04b3-dc76-4b89-8d28-3038652fd7bf","Type":"ContainerDied","Data":"56e4c08f9e42d3ea818b5e4c8fd1bcf4ed93ca026c3e47fb577c6e0a6e801432"}
Oct 30 00:35:37.460248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.460093    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.460248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.460106    6099 scope.go:122] "RemoveContainer" containerID="23deb9e025e27d9e7dbb292cd389ecc5fcb9c5371ebe72e02ecfabdcbde56444"
Oct 30 00:35:37.460331 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.460096    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"426a04b3-dc76-4b89-8d28-3038652fd7bf","Type":"ContainerDied","Data":"bf396d37b29bfa2006e3935fa69145a8acb684784ba731ede764600f0f477060"}
Oct 30 00:35:37.460384 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.460327    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"426a04b3-dc76-4b89-8d28-3038652fd7bf","Type":"ContainerDied","Data":"00ed8e3406e02fbd84b919cd0d099d011999035f9933db5cc62581e6107c63a1"}
Oct 30 00:35:37.468415 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.468399    6099 scope.go:122] "RemoveContainer" containerID="56e4c08f9e42d3ea818b5e4c8fd1bcf4ed93ca026c3e47fb577c6e0a6e801432"
Oct 30 00:35:37.479994 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.479977    6099 scope.go:122] "RemoveContainer" containerID="5515e80e8e28805e47ca2d5dd4127719c0b4fd426c264e0ec0becc2f84687f9d"
Oct 30 00:35:37.486351 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.486324    6099 scope.go:122] "RemoveContainer" containerID="bf396d37b29bfa2006e3935fa69145a8acb684784ba731ede764600f0f477060"
Oct 30 00:35:37.492902 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.492886    6099 scope.go:122] "RemoveContainer" containerID="6a69994adc0ccc20c20c18cc3c4b2db5e34a2175084c002010effc404a0b33b6"
Oct 30 00:35:37.499677 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.499660    6099 scope.go:122] "RemoveContainer" containerID="273b61e0df7f8e20c3a0d13e72a028b7f29b04f04826d53026d3244a3444304f"
Oct 30 00:35:37.506525 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.506434    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 00:35:37.506582 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.506552    6099 scope.go:122] "RemoveContainer" containerID="5be1dff3726a9bbaf208d7e7c745e2af4e1fb89d65e25c3dfb3564f5a9011269"
Oct 30 00:35:37.513320 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.513304    6099 scope.go:122] "RemoveContainer" containerID="23deb9e025e27d9e7dbb292cd389ecc5fcb9c5371ebe72e02ecfabdcbde56444"
Oct 30 00:35:37.513588 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:37.513568    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23deb9e025e27d9e7dbb292cd389ecc5fcb9c5371ebe72e02ecfabdcbde56444\": container with ID starting with 23deb9e025e27d9e7dbb292cd389ecc5fcb9c5371ebe72e02ecfabdcbde56444 not found: ID does not exist" containerID="23deb9e025e27d9e7dbb292cd389ecc5fcb9c5371ebe72e02ecfabdcbde56444"
Oct 30 00:35:37.513665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.513601    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23deb9e025e27d9e7dbb292cd389ecc5fcb9c5371ebe72e02ecfabdcbde56444"} err="failed to get container status \"23deb9e025e27d9e7dbb292cd389ecc5fcb9c5371ebe72e02ecfabdcbde56444\": rpc error: code = NotFound desc = could not find container \"23deb9e025e27d9e7dbb292cd389ecc5fcb9c5371ebe72e02ecfabdcbde56444\": container with ID starting with 23deb9e025e27d9e7dbb292cd389ecc5fcb9c5371ebe72e02ecfabdcbde56444 not found: ID does not exist"
Oct 30 00:35:37.513665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.513645    6099 scope.go:122] "RemoveContainer" containerID="56e4c08f9e42d3ea818b5e4c8fd1bcf4ed93ca026c3e47fb577c6e0a6e801432"
Oct 30 00:35:37.513909 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:37.513893    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e4c08f9e42d3ea818b5e4c8fd1bcf4ed93ca026c3e47fb577c6e0a6e801432\": container with ID starting with 56e4c08f9e42d3ea818b5e4c8fd1bcf4ed93ca026c3e47fb577c6e0a6e801432 not found: ID does not exist" containerID="56e4c08f9e42d3ea818b5e4c8fd1bcf4ed93ca026c3e47fb577c6e0a6e801432"
Oct 30 00:35:37.513977 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.513917    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e4c08f9e42d3ea818b5e4c8fd1bcf4ed93ca026c3e47fb577c6e0a6e801432"} err="failed to get container status \"56e4c08f9e42d3ea818b5e4c8fd1bcf4ed93ca026c3e47fb577c6e0a6e801432\": rpc error: code = NotFound desc = could not find container \"56e4c08f9e42d3ea818b5e4c8fd1bcf4ed93ca026c3e47fb577c6e0a6e801432\": container with ID starting with 56e4c08f9e42d3ea818b5e4c8fd1bcf4ed93ca026c3e47fb577c6e0a6e801432 not found: ID does not exist"
Oct 30 00:35:37.513977 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.513937    6099 scope.go:122] "RemoveContainer" containerID="5515e80e8e28805e47ca2d5dd4127719c0b4fd426c264e0ec0becc2f84687f9d"
Oct 30 00:35:37.514193 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:37.514175    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5515e80e8e28805e47ca2d5dd4127719c0b4fd426c264e0ec0becc2f84687f9d\": container with ID starting with 5515e80e8e28805e47ca2d5dd4127719c0b4fd426c264e0ec0becc2f84687f9d not found: ID does not exist" containerID="5515e80e8e28805e47ca2d5dd4127719c0b4fd426c264e0ec0becc2f84687f9d"
Oct 30 00:35:37.514234 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.514200    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5515e80e8e28805e47ca2d5dd4127719c0b4fd426c264e0ec0becc2f84687f9d"} err="failed to get container status \"5515e80e8e28805e47ca2d5dd4127719c0b4fd426c264e0ec0becc2f84687f9d\": rpc error: code = NotFound desc = could not find container \"5515e80e8e28805e47ca2d5dd4127719c0b4fd426c264e0ec0becc2f84687f9d\": container with ID starting with 5515e80e8e28805e47ca2d5dd4127719c0b4fd426c264e0ec0becc2f84687f9d not found: ID does not exist"
Oct 30 00:35:37.514234 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.514220    6099 scope.go:122] "RemoveContainer" containerID="bf396d37b29bfa2006e3935fa69145a8acb684784ba731ede764600f0f477060"
Oct 30 00:35:37.514425 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:37.514407    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf396d37b29bfa2006e3935fa69145a8acb684784ba731ede764600f0f477060\": container with ID starting with bf396d37b29bfa2006e3935fa69145a8acb684784ba731ede764600f0f477060 not found: ID does not exist" containerID="bf396d37b29bfa2006e3935fa69145a8acb684784ba731ede764600f0f477060"
Oct 30 00:35:37.514503 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.514433    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf396d37b29bfa2006e3935fa69145a8acb684784ba731ede764600f0f477060"} err="failed to get container status \"bf396d37b29bfa2006e3935fa69145a8acb684784ba731ede764600f0f477060\": rpc error: code = NotFound desc = could not find container \"bf396d37b29bfa2006e3935fa69145a8acb684784ba731ede764600f0f477060\": container with ID starting with bf396d37b29bfa2006e3935fa69145a8acb684784ba731ede764600f0f477060 not found: ID does not exist"
Oct 30 00:35:37.514503 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.514451    6099 scope.go:122] "RemoveContainer" containerID="6a69994adc0ccc20c20c18cc3c4b2db5e34a2175084c002010effc404a0b33b6"
Oct 30 00:35:37.514685 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:37.514667    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a69994adc0ccc20c20c18cc3c4b2db5e34a2175084c002010effc404a0b33b6\": container with ID starting with 6a69994adc0ccc20c20c18cc3c4b2db5e34a2175084c002010effc404a0b33b6 not found: ID does not exist" containerID="6a69994adc0ccc20c20c18cc3c4b2db5e34a2175084c002010effc404a0b33b6"
Oct 30 00:35:37.514782 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.514691    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a69994adc0ccc20c20c18cc3c4b2db5e34a2175084c002010effc404a0b33b6"} err="failed to get container status \"6a69994adc0ccc20c20c18cc3c4b2db5e34a2175084c002010effc404a0b33b6\": rpc error: code = NotFound desc = could not find container \"6a69994adc0ccc20c20c18cc3c4b2db5e34a2175084c002010effc404a0b33b6\": container with ID starting with 6a69994adc0ccc20c20c18cc3c4b2db5e34a2175084c002010effc404a0b33b6 not found: ID does not exist"
Oct 30 00:35:37.514848 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.514787    6099 scope.go:122] "RemoveContainer" containerID="273b61e0df7f8e20c3a0d13e72a028b7f29b04f04826d53026d3244a3444304f"
Oct 30 00:35:37.515034 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:37.515016    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273b61e0df7f8e20c3a0d13e72a028b7f29b04f04826d53026d3244a3444304f\": container with ID starting with 273b61e0df7f8e20c3a0d13e72a028b7f29b04f04826d53026d3244a3444304f not found: ID does not exist" containerID="273b61e0df7f8e20c3a0d13e72a028b7f29b04f04826d53026d3244a3444304f"
Oct 30 00:35:37.515095 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.515038    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273b61e0df7f8e20c3a0d13e72a028b7f29b04f04826d53026d3244a3444304f"} err="failed to get container status \"273b61e0df7f8e20c3a0d13e72a028b7f29b04f04826d53026d3244a3444304f\": rpc error: code = NotFound desc = could not find container \"273b61e0df7f8e20c3a0d13e72a028b7f29b04f04826d53026d3244a3444304f\": container with ID starting with 273b61e0df7f8e20c3a0d13e72a028b7f29b04f04826d53026d3244a3444304f not found: ID does not exist"
Oct 30 00:35:37.515095 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.515052    6099 scope.go:122] "RemoveContainer" containerID="5be1dff3726a9bbaf208d7e7c745e2af4e1fb89d65e25c3dfb3564f5a9011269"
Oct 30 00:35:37.515290 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:37.515272    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5be1dff3726a9bbaf208d7e7c745e2af4e1fb89d65e25c3dfb3564f5a9011269\": container with ID starting with 5be1dff3726a9bbaf208d7e7c745e2af4e1fb89d65e25c3dfb3564f5a9011269 not found: ID does not exist" containerID="5be1dff3726a9bbaf208d7e7c745e2af4e1fb89d65e25c3dfb3564f5a9011269"
Oct 30 00:35:37.515332 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.515295    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be1dff3726a9bbaf208d7e7c745e2af4e1fb89d65e25c3dfb3564f5a9011269"} err="failed to get container status \"5be1dff3726a9bbaf208d7e7c745e2af4e1fb89d65e25c3dfb3564f5a9011269\": rpc error: code = NotFound desc = could not find container \"5be1dff3726a9bbaf208d7e7c745e2af4e1fb89d65e25c3dfb3564f5a9011269\": container with ID starting with 5be1dff3726a9bbaf208d7e7c745e2af4e1fb89d65e25c3dfb3564f5a9011269 not found: ID does not exist"
Oct 30 00:35:37.515332 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.515308    6099 scope.go:122] "RemoveContainer" containerID="23deb9e025e27d9e7dbb292cd389ecc5fcb9c5371ebe72e02ecfabdcbde56444"
Oct 30 00:35:37.515528 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.515509    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23deb9e025e27d9e7dbb292cd389ecc5fcb9c5371ebe72e02ecfabdcbde56444"} err="failed to get container status \"23deb9e025e27d9e7dbb292cd389ecc5fcb9c5371ebe72e02ecfabdcbde56444\": rpc error: code = NotFound desc = could not find container \"23deb9e025e27d9e7dbb292cd389ecc5fcb9c5371ebe72e02ecfabdcbde56444\": container with ID starting with 23deb9e025e27d9e7dbb292cd389ecc5fcb9c5371ebe72e02ecfabdcbde56444 not found: ID does not exist"
Oct 30 00:35:37.515589 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.515529    6099 scope.go:122] "RemoveContainer" containerID="56e4c08f9e42d3ea818b5e4c8fd1bcf4ed93ca026c3e47fb577c6e0a6e801432"
Oct 30 00:35:37.515805 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.515782    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e4c08f9e42d3ea818b5e4c8fd1bcf4ed93ca026c3e47fb577c6e0a6e801432"} err="failed to get container status \"56e4c08f9e42d3ea818b5e4c8fd1bcf4ed93ca026c3e47fb577c6e0a6e801432\": rpc error: code = NotFound desc = could not find container \"56e4c08f9e42d3ea818b5e4c8fd1bcf4ed93ca026c3e47fb577c6e0a6e801432\": container with ID starting with 56e4c08f9e42d3ea818b5e4c8fd1bcf4ed93ca026c3e47fb577c6e0a6e801432 not found: ID does not exist"
Oct 30 00:35:37.515805 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.515802    6099 scope.go:122] "RemoveContainer" containerID="5515e80e8e28805e47ca2d5dd4127719c0b4fd426c264e0ec0becc2f84687f9d"
Oct 30 00:35:37.516022 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.516005    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5515e80e8e28805e47ca2d5dd4127719c0b4fd426c264e0ec0becc2f84687f9d"} err="failed to get container status \"5515e80e8e28805e47ca2d5dd4127719c0b4fd426c264e0ec0becc2f84687f9d\": rpc error: code = NotFound desc = could not find container \"5515e80e8e28805e47ca2d5dd4127719c0b4fd426c264e0ec0becc2f84687f9d\": container with ID starting with 5515e80e8e28805e47ca2d5dd4127719c0b4fd426c264e0ec0becc2f84687f9d not found: ID does not exist"
Oct 30 00:35:37.516022 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.516021    6099 scope.go:122] "RemoveContainer" containerID="bf396d37b29bfa2006e3935fa69145a8acb684784ba731ede764600f0f477060"
Oct 30 00:35:37.516221 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.516205    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf396d37b29bfa2006e3935fa69145a8acb684784ba731ede764600f0f477060"} err="failed to get container status \"bf396d37b29bfa2006e3935fa69145a8acb684784ba731ede764600f0f477060\": rpc error: code = NotFound desc = could not find container \"bf396d37b29bfa2006e3935fa69145a8acb684784ba731ede764600f0f477060\": container with ID starting with bf396d37b29bfa2006e3935fa69145a8acb684784ba731ede764600f0f477060 not found: ID does not exist"
Oct 30 00:35:37.516221 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.516221    6099 scope.go:122] "RemoveContainer" containerID="6a69994adc0ccc20c20c18cc3c4b2db5e34a2175084c002010effc404a0b33b6"
Oct 30 00:35:37.516431 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.516415    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a69994adc0ccc20c20c18cc3c4b2db5e34a2175084c002010effc404a0b33b6"} err="failed to get container status \"6a69994adc0ccc20c20c18cc3c4b2db5e34a2175084c002010effc404a0b33b6\": rpc error: code = NotFound desc = could not find container \"6a69994adc0ccc20c20c18cc3c4b2db5e34a2175084c002010effc404a0b33b6\": container with ID starting with 6a69994adc0ccc20c20c18cc3c4b2db5e34a2175084c002010effc404a0b33b6 not found: ID does not exist"
Oct 30 00:35:37.516509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.516431    6099 scope.go:122] "RemoveContainer" containerID="273b61e0df7f8e20c3a0d13e72a028b7f29b04f04826d53026d3244a3444304f"
Oct 30 00:35:37.516655 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.516634    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273b61e0df7f8e20c3a0d13e72a028b7f29b04f04826d53026d3244a3444304f"} err="failed to get container status \"273b61e0df7f8e20c3a0d13e72a028b7f29b04f04826d53026d3244a3444304f\": rpc error: code = NotFound desc = could not find container \"273b61e0df7f8e20c3a0d13e72a028b7f29b04f04826d53026d3244a3444304f\": container with ID starting with 273b61e0df7f8e20c3a0d13e72a028b7f29b04f04826d53026d3244a3444304f not found: ID does not exist"
Oct 30 00:35:37.516731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.516654    6099 scope.go:122] "RemoveContainer" containerID="5be1dff3726a9bbaf208d7e7c745e2af4e1fb89d65e25c3dfb3564f5a9011269"
Oct 30 00:35:37.516731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.516669    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 00:35:37.516867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.516850    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be1dff3726a9bbaf208d7e7c745e2af4e1fb89d65e25c3dfb3564f5a9011269"} err="failed to get container status \"5be1dff3726a9bbaf208d7e7c745e2af4e1fb89d65e25c3dfb3564f5a9011269\": rpc error: code = NotFound desc = could not find container \"5be1dff3726a9bbaf208d7e7c745e2af4e1fb89d65e25c3dfb3564f5a9011269\": container with ID starting with 5be1dff3726a9bbaf208d7e7c745e2af4e1fb89d65e25c3dfb3564f5a9011269 not found: ID does not exist"
Oct 30 00:35:37.541881 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.541856    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 00:35:37.542277 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542261    6099 cpu_manager.go:414] "containerMap: removing container" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="kube-rbac-proxy"
Oct 30 00:35:37.542397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542279    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="kube-rbac-proxy"
Oct 30 00:35:37.542397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542305    6099 cpu_manager.go:414] "containerMap: removing container" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="init-config-reloader"
Oct 30 00:35:37.542397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542312    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="init-config-reloader"
Oct 30 00:35:37.542397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542327    6099 cpu_manager.go:414] "containerMap: removing container" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="prom-label-proxy"
Oct 30 00:35:37.542397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542332    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="prom-label-proxy"
Oct 30 00:35:37.542397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542339    6099 cpu_manager.go:414] "containerMap: removing container" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="alertmanager"
Oct 30 00:35:37.542397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542343    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="alertmanager"
Oct 30 00:35:37.542397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542352    6099 cpu_manager.go:414] "containerMap: removing container" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="kube-rbac-proxy-metric"
Oct 30 00:35:37.542397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542358    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="kube-rbac-proxy-metric"
Oct 30 00:35:37.542397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542377    6099 cpu_manager.go:414] "containerMap: removing container" podUID="fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" containerName="console"
Oct 30 00:35:37.542397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542384    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" containerName="console"
Oct 30 00:35:37.542397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542396    6099 cpu_manager.go:414] "containerMap: removing container" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="kube-rbac-proxy-web"
Oct 30 00:35:37.542397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542400    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="kube-rbac-proxy-web"
Oct 30 00:35:37.542931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542410    6099 cpu_manager.go:414] "containerMap: removing container" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="config-reloader"
Oct 30 00:35:37.542931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542415    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="config-reloader"
Oct 30 00:35:37.542931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542475    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb1fcdd5-b4bc-436f-9350-cfe138b3d9bd" containerName="console"
Oct 30 00:35:37.542931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542484    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="kube-rbac-proxy-web"
Oct 30 00:35:37.542931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542491    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="config-reloader"
Oct 30 00:35:37.542931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542496    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="kube-rbac-proxy"
Oct 30 00:35:37.542931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542503    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="init-config-reloader"
Oct 30 00:35:37.542931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542509    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="prom-label-proxy"
Oct 30 00:35:37.542931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542514    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="alertmanager"
Oct 30 00:35:37.542931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.542520    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" containerName="kube-rbac-proxy-metric"
Oct 30 00:35:37.577169 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.577150    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 00:35:37.577310 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.577293    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.590648 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.590491    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 00:35:37.590648 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.590578    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 00:35:37.590648 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.590616    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 00:35:37.590884 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.590690    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 00:35:37.590884 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.590783    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 00:35:37.590884 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.590806    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 00:35:37.590884 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.590860    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-82whn\""
Oct 30 00:35:37.591024 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.590901    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 00:35:37.591123 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.591110    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 00:35:37.600809 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.600789    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 00:35:37.672940 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.672866    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ff948dc-8d7a-435f-83bd-7a6a461c832a-config-out\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.672940 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.672906    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msq2w\" (UniqueName: \"kubernetes.io/projected/8ff948dc-8d7a-435f-83bd-7a6a461c832a-kube-api-access-msq2w\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.672940 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.672932    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-web-config\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.673122 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.673067    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.673122 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.673103    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.673222 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.673126    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.673222 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.673212    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ff948dc-8d7a-435f-83bd-7a6a461c832a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.673295 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.673247    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-config-volume\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.673332 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.673293    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8ff948dc-8d7a-435f-83bd-7a6a461c832a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.673332 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.673322    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ff948dc-8d7a-435f-83bd-7a6a461c832a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.673415 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.673339    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ff948dc-8d7a-435f-83bd-7a6a461c832a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.673415 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.673368    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.673488 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.673440    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.774529 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.774503    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ff948dc-8d7a-435f-83bd-7a6a461c832a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.774658 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.774532    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ff948dc-8d7a-435f-83bd-7a6a461c832a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.774658 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.774551    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.774658 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.774574    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.774658 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.774598    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ff948dc-8d7a-435f-83bd-7a6a461c832a-config-out\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.774903 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.774699    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msq2w\" (UniqueName: \"kubernetes.io/projected/8ff948dc-8d7a-435f-83bd-7a6a461c832a-kube-api-access-msq2w\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.774903 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.774764    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-web-config\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.774903 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.774791    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.774903 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.774823    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.774903 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.774856    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.774903 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.774882    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ff948dc-8d7a-435f-83bd-7a6a461c832a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.775251 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.774907    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-config-volume\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.775251 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.774957    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8ff948dc-8d7a-435f-83bd-7a6a461c832a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.775251 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.775094    6099 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 00:35:37.775412 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.775269    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8ff948dc-8d7a-435f-83bd-7a6a461c832a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.775468 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.775438    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ff948dc-8d7a-435f-83bd-7a6a461c832a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.776296 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.776266    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ff948dc-8d7a-435f-83bd-7a6a461c832a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.778054 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.777960    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.778054 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.777967    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-web-config\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.778054 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.778005    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ff948dc-8d7a-435f-83bd-7a6a461c832a-config-out\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.778054 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.778036    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ff948dc-8d7a-435f-83bd-7a6a461c832a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.778299 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.778101    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-config-volume\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.778299 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.778227    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.778299 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.778231    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.778836 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.778816    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.779635 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.779616    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ff948dc-8d7a-435f-83bd-7a6a461c832a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.795340 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.795318    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msq2w\" (UniqueName: \"kubernetes.io/projected/8ff948dc-8d7a-435f-83bd-7a6a461c832a-kube-api-access-msq2w\") pod \"alertmanager-main-0\" (UID: \"8ff948dc-8d7a-435f-83bd-7a6a461c832a\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:37.970334 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:37.970260    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:35:38.262051 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:38.260667    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="426a04b3-dc76-4b89-8d28-3038652fd7bf" path="/var/lib/kubelet/pods/426a04b3-dc76-4b89-8d28-3038652fd7bf/volumes"
Oct 30 00:35:38.317207 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:38.317177    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 00:35:38.320942 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:38.320911    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff948dc_8d7a_435f_83bd_7a6a461c832a.slice/crio-ec409b32d1ca03b74b19b59f7ba9b4e368c6acc552c7d09428506d9e3eed7d41 WatchSource:0}: Error finding container ec409b32d1ca03b74b19b59f7ba9b4e368c6acc552c7d09428506d9e3eed7d41: Status 404 returned error can't find the container with id ec409b32d1ca03b74b19b59f7ba9b4e368c6acc552c7d09428506d9e3eed7d41
Oct 30 00:35:38.465345 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:38.465317    6099 generic.go:356] "Generic (PLEG): container finished" podID="8ff948dc-8d7a-435f-83bd-7a6a461c832a" containerID="ce66a30078120cfc1035d56feaed10c8f44b1565a4b2e201effad773d1f3c066" exitCode=0
Oct 30 00:35:38.465476 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:38.465399    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8ff948dc-8d7a-435f-83bd-7a6a461c832a","Type":"ContainerDied","Data":"ce66a30078120cfc1035d56feaed10c8f44b1565a4b2e201effad773d1f3c066"}
Oct 30 00:35:38.465476 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:38.465458    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8ff948dc-8d7a-435f-83bd-7a6a461c832a","Type":"ContainerStarted","Data":"ec409b32d1ca03b74b19b59f7ba9b4e368c6acc552c7d09428506d9e3eed7d41"}
Oct 30 00:35:39.475356 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:39.475325    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8ff948dc-8d7a-435f-83bd-7a6a461c832a","Type":"ContainerStarted","Data":"fe36fa6b58fc105cdadc2e2e1d6936702e8cb5968a3742cb1b81534f7e1087aa"}
Oct 30 00:35:39.475949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:39.475398    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8ff948dc-8d7a-435f-83bd-7a6a461c832a","Type":"ContainerStarted","Data":"e7ccf43b625e69f52491a6257234771e29d03aaa07ea5d2cd64267f757baed3a"}
Oct 30 00:35:39.475949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:39.475445    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8ff948dc-8d7a-435f-83bd-7a6a461c832a","Type":"ContainerStarted","Data":"8a3ab1bc9be9637db6aa75c99f10a00a21fafb2a27224932746e42e885f564a9"}
Oct 30 00:35:39.475949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:39.475488    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8ff948dc-8d7a-435f-83bd-7a6a461c832a","Type":"ContainerStarted","Data":"c40f32526fde9a8e7ce521a9baec5bc4bb7b48076127a572cffe70865b441cc5"}
Oct 30 00:35:39.475949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:39.475529    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8ff948dc-8d7a-435f-83bd-7a6a461c832a","Type":"ContainerStarted","Data":"c60a811e36d95c0403d2dbe5bdc087f73f56ef271b04175b0949b1d48fe16339"}
Oct 30 00:35:39.475949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:39.475572    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8ff948dc-8d7a-435f-83bd-7a6a461c832a","Type":"ContainerStarted","Data":"8bae08d6771c7a9bfeec59e52550ce9d1619edc226728da08ac55e8d8aaadb74"}
Oct 30 00:35:39.526144 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:39.526098    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.513134148 podStartE2EDuration="2.526081947s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.947799ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:39.525317727 +0000 UTC m=+33.382225481" watchObservedRunningTime="2026-10-30 00:35:39.526081947 +0000 UTC m=+33.382989693"
Oct 30 00:35:40.277518 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:40.276645    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:35:40.277518 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:40.277020    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="prometheus" containerID="cri-o://4e74e8f29a1a34981cd8627fe553677682337f9869b8d7ed78dccf8dc2ef7271" gracePeriod=600
Oct 30 00:35:40.277518 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:40.277053    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="kube-rbac-proxy" containerID="cri-o://3f5f67376608b0f94747220f6a0557fe980a6367187405145ca6b992c1b0bf4c" gracePeriod=600
Oct 30 00:35:40.277518 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:40.277263    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="kube-rbac-proxy-thanos" containerID="cri-o://bbf9954a9960660aab5cc5a2c2c53899fcc2550f9e97de9917b1ddbe65246f62" gracePeriod=600
Oct 30 00:35:40.277518 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:40.277306    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="kube-rbac-proxy-web" containerID="cri-o://0c22a82ec34c5be2bdb4330d03eafa5e2924a2600f436618fed37ee98885eab8" gracePeriod=600
Oct 30 00:35:40.277518 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:40.277385    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="thanos-sidecar" containerID="cri-o://4000241637e9c13b529d8b826baffde60b8ede2175e48dcb09f5941ac1fd7e42" gracePeriod=600
Oct 30 00:35:40.277518 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:40.277386    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="config-reloader" containerID="cri-o://061004aa234d1293a1a255e1c7f07cbba3e09f44d4799f45a066ca1b18ef5aa3" gracePeriod=600
Oct 30 00:35:40.279565 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:40.278916    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-62gpq"
Oct 30 00:35:40.486438 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:40.486391    6099 generic.go:356] "Generic (PLEG): container finished" podID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerID="bbf9954a9960660aab5cc5a2c2c53899fcc2550f9e97de9917b1ddbe65246f62" exitCode=0
Oct 30 00:35:40.486438 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:40.486421    6099 generic.go:356] "Generic (PLEG): container finished" podID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerID="3f5f67376608b0f94747220f6a0557fe980a6367187405145ca6b992c1b0bf4c" exitCode=0
Oct 30 00:35:40.486438 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:40.486428    6099 generic.go:356] "Generic (PLEG): container finished" podID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerID="4000241637e9c13b529d8b826baffde60b8ede2175e48dcb09f5941ac1fd7e42" exitCode=0
Oct 30 00:35:40.486438 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:40.486434    6099 generic.go:356] "Generic (PLEG): container finished" podID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerID="061004aa234d1293a1a255e1c7f07cbba3e09f44d4799f45a066ca1b18ef5aa3" exitCode=0
Oct 30 00:35:40.486438 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:40.486439    6099 generic.go:356] "Generic (PLEG): container finished" podID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerID="4e74e8f29a1a34981cd8627fe553677682337f9869b8d7ed78dccf8dc2ef7271" exitCode=0
Oct 30 00:35:40.487020 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:40.486463    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026218e5-d0ac-48b3-9023-f245d5cf9848","Type":"ContainerDied","Data":"bbf9954a9960660aab5cc5a2c2c53899fcc2550f9e97de9917b1ddbe65246f62"}
Oct 30 00:35:40.487020 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:40.486580    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026218e5-d0ac-48b3-9023-f245d5cf9848","Type":"ContainerDied","Data":"3f5f67376608b0f94747220f6a0557fe980a6367187405145ca6b992c1b0bf4c"}
Oct 30 00:35:40.487020 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:40.486624    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026218e5-d0ac-48b3-9023-f245d5cf9848","Type":"ContainerDied","Data":"4000241637e9c13b529d8b826baffde60b8ede2175e48dcb09f5941ac1fd7e42"}
Oct 30 00:35:40.487020 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:40.486664    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026218e5-d0ac-48b3-9023-f245d5cf9848","Type":"ContainerDied","Data":"061004aa234d1293a1a255e1c7f07cbba3e09f44d4799f45a066ca1b18ef5aa3"}
Oct 30 00:35:40.487020 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:40.486717    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026218e5-d0ac-48b3-9023-f245d5cf9848","Type":"ContainerDied","Data":"4e74e8f29a1a34981cd8627fe553677682337f9869b8d7ed78dccf8dc2ef7271"}
Oct 30 00:35:41.620525 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.620503    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:41.711086 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711021    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-web-config\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-web-config\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.711086 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711057    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-config-out\" (UniqueName: \"kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-config-out\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.711294 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711088    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-config\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-config\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.711294 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711110    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-trusted-ca-bundle\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.711294 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711140    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-metrics-client-ca\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.711294 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711174    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-kube-api-access-j5kb7\" (UniqueName: \"kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-kube-api-access-j5kb7\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.711294 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711201    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-kube-rbac-proxy\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.711294 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711231    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.711294 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711265    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-db\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.711294 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711294    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-grpc-tls\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.711682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711342    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-tls-assets\" (UniqueName: \"kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-tls-assets\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.711682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711373    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-1\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.711682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711419    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-2\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.711682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711459    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-thanos-prometheus-http-client-file\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.711682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711498    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-metrics-client-certs\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.711682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711537    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-kubelet-serving-ca-bundle\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.711682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711568    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-tls\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.711682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711603    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-0\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.711682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711640    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-serving-certs-ca-bundle\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.712144 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.711687    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"026218e5-d0ac-48b3-9023-f245d5cf9848\" (UID: \"026218e5-d0ac-48b3-9023-f245d5cf9848\") "
Oct 30 00:35:41.714444 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.714411    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-kubelet-serving-ca-bundle" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:35:41.714573 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.714518    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-db" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:35:41.715699 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.715672    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-serving-certs-ca-bundle" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:35:41.715831 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.715811    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-kube-rbac-proxy-web" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:41.715902 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.715814    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-1" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:35:41.716008 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.715985    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-grpc-tls" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:41.716172 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.716154    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-2" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:35:41.716967 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.716308    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-kube-rbac-proxy" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:41.716967 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.716365    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-config-out" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:35:41.716967 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.716653    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-0" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:35:41.716967 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.716758    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-metrics-client-ca" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:35:41.716967 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.716821    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-trusted-ca-bundle" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:35:41.718196 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.718171    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-tls" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:41.719178 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.719153    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-kube-api-access-j5kb7" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "kube-api-access-j5kb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:35:41.719258 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.719239    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-tls-assets" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:35:41.720316 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.720282    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-config" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:41.720386 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.720347    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-metrics-client-certs" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:41.720578 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.720556    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-thanos-prometheus-http-client-file" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:41.721526 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.721504    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-thanos-sidecar-tls" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:41.727789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.727767    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-web-config" pod "026218e5-d0ac-48b3-9023-f245d5cf9848" (UID: "026218e5-d0ac-48b3-9023-f245d5cf9848"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:35:41.813029 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813001    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5kb7\" (UniqueName: \"kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-kube-api-access-j5kb7\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813029 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813024    6099 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-kube-rbac-proxy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813042    6099 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813053    6099 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-db\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813062    6099 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-grpc-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813070    6099 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/026218e5-d0ac-48b3-9023-f245d5cf9848-tls-assets\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813078    6099 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813087    6099 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813096    6099 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813104    6099 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-metrics-client-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813112    6099 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813121    6099 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813129    6099 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813139    6099 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813147    6099 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813155    6099 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-web-config\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813163    6099 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/026218e5-d0ac-48b3-9023-f245d5cf9848-config-out\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813171    6099 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/026218e5-d0ac-48b3-9023-f245d5cf9848-config\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813178    6099 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:41.813191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:41.813186    6099 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/026218e5-d0ac-48b3-9023-f245d5cf9848-configmap-metrics-client-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:35:42.497403 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.497373    6099 generic.go:356] "Generic (PLEG): container finished" podID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerID="0c22a82ec34c5be2bdb4330d03eafa5e2924a2600f436618fed37ee98885eab8" exitCode=0
Oct 30 00:35:42.497573 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.497453    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.497573 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.497449    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026218e5-d0ac-48b3-9023-f245d5cf9848","Type":"ContainerDied","Data":"0c22a82ec34c5be2bdb4330d03eafa5e2924a2600f436618fed37ee98885eab8"}
Oct 30 00:35:42.497573 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.497539    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"026218e5-d0ac-48b3-9023-f245d5cf9848","Type":"ContainerDied","Data":"d9ad9b60e9424a59c50647f75bc4c95b224255b4e983bf3a664f5b06e50e507c"}
Oct 30 00:35:42.497755 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.497587    6099 scope.go:122] "RemoveContainer" containerID="bbf9954a9960660aab5cc5a2c2c53899fcc2550f9e97de9917b1ddbe65246f62"
Oct 30 00:35:42.505127 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.505108    6099 scope.go:122] "RemoveContainer" containerID="3f5f67376608b0f94747220f6a0557fe980a6367187405145ca6b992c1b0bf4c"
Oct 30 00:35:42.511936 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.511918    6099 scope.go:122] "RemoveContainer" containerID="0c22a82ec34c5be2bdb4330d03eafa5e2924a2600f436618fed37ee98885eab8"
Oct 30 00:35:42.518797 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.518782    6099 scope.go:122] "RemoveContainer" containerID="4000241637e9c13b529d8b826baffde60b8ede2175e48dcb09f5941ac1fd7e42"
Oct 30 00:35:42.525071 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.525055    6099 scope.go:122] "RemoveContainer" containerID="061004aa234d1293a1a255e1c7f07cbba3e09f44d4799f45a066ca1b18ef5aa3"
Oct 30 00:35:42.531470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.531450    6099 scope.go:122] "RemoveContainer" containerID="4e74e8f29a1a34981cd8627fe553677682337f9869b8d7ed78dccf8dc2ef7271"
Oct 30 00:35:42.538328 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.538313    6099 scope.go:122] "RemoveContainer" containerID="289fa18bc40491e3b05eba723fa2e20720972feb42138c2f8902d6a3f206dc45"
Oct 30 00:35:42.544563 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.544544    6099 scope.go:122] "RemoveContainer" containerID="bbf9954a9960660aab5cc5a2c2c53899fcc2550f9e97de9917b1ddbe65246f62"
Oct 30 00:35:42.544844 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:42.544822    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbf9954a9960660aab5cc5a2c2c53899fcc2550f9e97de9917b1ddbe65246f62\": container with ID starting with bbf9954a9960660aab5cc5a2c2c53899fcc2550f9e97de9917b1ddbe65246f62 not found: ID does not exist" containerID="bbf9954a9960660aab5cc5a2c2c53899fcc2550f9e97de9917b1ddbe65246f62"
Oct 30 00:35:42.544908 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.544854    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbf9954a9960660aab5cc5a2c2c53899fcc2550f9e97de9917b1ddbe65246f62"} err="failed to get container status \"bbf9954a9960660aab5cc5a2c2c53899fcc2550f9e97de9917b1ddbe65246f62\": rpc error: code = NotFound desc = could not find container \"bbf9954a9960660aab5cc5a2c2c53899fcc2550f9e97de9917b1ddbe65246f62\": container with ID starting with bbf9954a9960660aab5cc5a2c2c53899fcc2550f9e97de9917b1ddbe65246f62 not found: ID does not exist"
Oct 30 00:35:42.544908 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.544871    6099 scope.go:122] "RemoveContainer" containerID="3f5f67376608b0f94747220f6a0557fe980a6367187405145ca6b992c1b0bf4c"
Oct 30 00:35:42.545151 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:42.545112    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f5f67376608b0f94747220f6a0557fe980a6367187405145ca6b992c1b0bf4c\": container with ID starting with 3f5f67376608b0f94747220f6a0557fe980a6367187405145ca6b992c1b0bf4c not found: ID does not exist" containerID="3f5f67376608b0f94747220f6a0557fe980a6367187405145ca6b992c1b0bf4c"
Oct 30 00:35:42.545219 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.545159    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f5f67376608b0f94747220f6a0557fe980a6367187405145ca6b992c1b0bf4c"} err="failed to get container status \"3f5f67376608b0f94747220f6a0557fe980a6367187405145ca6b992c1b0bf4c\": rpc error: code = NotFound desc = could not find container \"3f5f67376608b0f94747220f6a0557fe980a6367187405145ca6b992c1b0bf4c\": container with ID starting with 3f5f67376608b0f94747220f6a0557fe980a6367187405145ca6b992c1b0bf4c not found: ID does not exist"
Oct 30 00:35:42.545219 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.545180    6099 scope.go:122] "RemoveContainer" containerID="0c22a82ec34c5be2bdb4330d03eafa5e2924a2600f436618fed37ee98885eab8"
Oct 30 00:35:42.545432 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:42.545412    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c22a82ec34c5be2bdb4330d03eafa5e2924a2600f436618fed37ee98885eab8\": container with ID starting with 0c22a82ec34c5be2bdb4330d03eafa5e2924a2600f436618fed37ee98885eab8 not found: ID does not exist" containerID="0c22a82ec34c5be2bdb4330d03eafa5e2924a2600f436618fed37ee98885eab8"
Oct 30 00:35:42.545471 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.545438    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c22a82ec34c5be2bdb4330d03eafa5e2924a2600f436618fed37ee98885eab8"} err="failed to get container status \"0c22a82ec34c5be2bdb4330d03eafa5e2924a2600f436618fed37ee98885eab8\": rpc error: code = NotFound desc = could not find container \"0c22a82ec34c5be2bdb4330d03eafa5e2924a2600f436618fed37ee98885eab8\": container with ID starting with 0c22a82ec34c5be2bdb4330d03eafa5e2924a2600f436618fed37ee98885eab8 not found: ID does not exist"
Oct 30 00:35:42.545471 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.545452    6099 scope.go:122] "RemoveContainer" containerID="4000241637e9c13b529d8b826baffde60b8ede2175e48dcb09f5941ac1fd7e42"
Oct 30 00:35:42.545691 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:42.545675    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4000241637e9c13b529d8b826baffde60b8ede2175e48dcb09f5941ac1fd7e42\": container with ID starting with 4000241637e9c13b529d8b826baffde60b8ede2175e48dcb09f5941ac1fd7e42 not found: ID does not exist" containerID="4000241637e9c13b529d8b826baffde60b8ede2175e48dcb09f5941ac1fd7e42"
Oct 30 00:35:42.546043 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.545695    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4000241637e9c13b529d8b826baffde60b8ede2175e48dcb09f5941ac1fd7e42"} err="failed to get container status \"4000241637e9c13b529d8b826baffde60b8ede2175e48dcb09f5941ac1fd7e42\": rpc error: code = NotFound desc = could not find container \"4000241637e9c13b529d8b826baffde60b8ede2175e48dcb09f5941ac1fd7e42\": container with ID starting with 4000241637e9c13b529d8b826baffde60b8ede2175e48dcb09f5941ac1fd7e42 not found: ID does not exist"
Oct 30 00:35:42.546043 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.545723    6099 scope.go:122] "RemoveContainer" containerID="061004aa234d1293a1a255e1c7f07cbba3e09f44d4799f45a066ca1b18ef5aa3"
Oct 30 00:35:42.546043 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:42.545926    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061004aa234d1293a1a255e1c7f07cbba3e09f44d4799f45a066ca1b18ef5aa3\": container with ID starting with 061004aa234d1293a1a255e1c7f07cbba3e09f44d4799f45a066ca1b18ef5aa3 not found: ID does not exist" containerID="061004aa234d1293a1a255e1c7f07cbba3e09f44d4799f45a066ca1b18ef5aa3"
Oct 30 00:35:42.546043 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.545941    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061004aa234d1293a1a255e1c7f07cbba3e09f44d4799f45a066ca1b18ef5aa3"} err="failed to get container status \"061004aa234d1293a1a255e1c7f07cbba3e09f44d4799f45a066ca1b18ef5aa3\": rpc error: code = NotFound desc = could not find container \"061004aa234d1293a1a255e1c7f07cbba3e09f44d4799f45a066ca1b18ef5aa3\": container with ID starting with 061004aa234d1293a1a255e1c7f07cbba3e09f44d4799f45a066ca1b18ef5aa3 not found: ID does not exist"
Oct 30 00:35:42.546043 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.545955    6099 scope.go:122] "RemoveContainer" containerID="4e74e8f29a1a34981cd8627fe553677682337f9869b8d7ed78dccf8dc2ef7271"
Oct 30 00:35:42.546267 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:42.546200    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e74e8f29a1a34981cd8627fe553677682337f9869b8d7ed78dccf8dc2ef7271\": container with ID starting with 4e74e8f29a1a34981cd8627fe553677682337f9869b8d7ed78dccf8dc2ef7271 not found: ID does not exist" containerID="4e74e8f29a1a34981cd8627fe553677682337f9869b8d7ed78dccf8dc2ef7271"
Oct 30 00:35:42.546267 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.546216    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e74e8f29a1a34981cd8627fe553677682337f9869b8d7ed78dccf8dc2ef7271"} err="failed to get container status \"4e74e8f29a1a34981cd8627fe553677682337f9869b8d7ed78dccf8dc2ef7271\": rpc error: code = NotFound desc = could not find container \"4e74e8f29a1a34981cd8627fe553677682337f9869b8d7ed78dccf8dc2ef7271\": container with ID starting with 4e74e8f29a1a34981cd8627fe553677682337f9869b8d7ed78dccf8dc2ef7271 not found: ID does not exist"
Oct 30 00:35:42.546267 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.546229    6099 scope.go:122] "RemoveContainer" containerID="289fa18bc40491e3b05eba723fa2e20720972feb42138c2f8902d6a3f206dc45"
Oct 30 00:35:42.546479 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:42.546437    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"289fa18bc40491e3b05eba723fa2e20720972feb42138c2f8902d6a3f206dc45\": container with ID starting with 289fa18bc40491e3b05eba723fa2e20720972feb42138c2f8902d6a3f206dc45 not found: ID does not exist" containerID="289fa18bc40491e3b05eba723fa2e20720972feb42138c2f8902d6a3f206dc45"
Oct 30 00:35:42.546479 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.546452    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"289fa18bc40491e3b05eba723fa2e20720972feb42138c2f8902d6a3f206dc45"} err="failed to get container status \"289fa18bc40491e3b05eba723fa2e20720972feb42138c2f8902d6a3f206dc45\": rpc error: code = NotFound desc = could not find container \"289fa18bc40491e3b05eba723fa2e20720972feb42138c2f8902d6a3f206dc45\": container with ID starting with 289fa18bc40491e3b05eba723fa2e20720972feb42138c2f8902d6a3f206dc45 not found: ID does not exist"
Oct 30 00:35:42.555106 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.555084    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:35:42.564950 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.564932    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:35:42.589879 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.589860    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:35:42.590225 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590210    6099 cpu_manager.go:414] "containerMap: removing container" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="init-config-reloader"
Oct 30 00:35:42.590278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590226    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="init-config-reloader"
Oct 30 00:35:42.590278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590247    6099 cpu_manager.go:414] "containerMap: removing container" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="kube-rbac-proxy-thanos"
Oct 30 00:35:42.590278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590253    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="kube-rbac-proxy-thanos"
Oct 30 00:35:42.590278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590261    6099 cpu_manager.go:414] "containerMap: removing container" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="kube-rbac-proxy"
Oct 30 00:35:42.590278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590265    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="kube-rbac-proxy"
Oct 30 00:35:42.590278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590272    6099 cpu_manager.go:414] "containerMap: removing container" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="kube-rbac-proxy-web"
Oct 30 00:35:42.590278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590277    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="kube-rbac-proxy-web"
Oct 30 00:35:42.590470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590286    6099 cpu_manager.go:414] "containerMap: removing container" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="thanos-sidecar"
Oct 30 00:35:42.590470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590290    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="thanos-sidecar"
Oct 30 00:35:42.590470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590299    6099 cpu_manager.go:414] "containerMap: removing container" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="config-reloader"
Oct 30 00:35:42.590470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590303    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="config-reloader"
Oct 30 00:35:42.590470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590309    6099 cpu_manager.go:414] "containerMap: removing container" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="prometheus"
Oct 30 00:35:42.590470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590313    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="prometheus"
Oct 30 00:35:42.590470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590359    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="init-config-reloader"
Oct 30 00:35:42.590470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590369    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="kube-rbac-proxy-thanos"
Oct 30 00:35:42.590470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590375    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="kube-rbac-proxy"
Oct 30 00:35:42.590470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590380    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="kube-rbac-proxy-web"
Oct 30 00:35:42.590470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590386    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="thanos-sidecar"
Oct 30 00:35:42.590470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590392    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="config-reloader"
Oct 30 00:35:42.590470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.590399    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" containerName="prometheus"
Oct 30 00:35:42.594221 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.594204    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.610183 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.610159    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 00:35:42.610183 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.610174    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 00:35:42.610382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.610240    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 00:35:42.610382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.610286    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 00:35:42.610382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.610314    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7f0anuf4jt4pi\""
Oct 30 00:35:42.610382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.610322    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 00:35:42.610382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.610326    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hxlc4\""
Oct 30 00:35:42.610382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.610338    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 00:35:42.610382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.610359    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 00:35:42.610773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.610402    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 00:35:42.610773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.610539    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 00:35:42.610773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.610590    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 00:35:42.610773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.610692    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 00:35:42.610963 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.610855    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 00:35:42.621398 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.621378    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 00:35:42.621664 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.621576    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 00:35:42.625095 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.625073    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:35:42.719321 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719291    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ab060b7-ab03-4bd9-b917-9a4945a0b614-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.719321 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719325    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.719512 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719346    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.719512 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719397    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-config\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.719512 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719510    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.719621 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719528    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.719621 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719552    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.719621 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719582    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7ab060b7-ab03-4bd9-b917-9a4945a0b614-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.719753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719631    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.719753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719648    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.719753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719681    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.719753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719699    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ab060b7-ab03-4bd9-b917-9a4945a0b614-config-out\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.719895 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719763    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.719895 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719779    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwtx8\" (UniqueName: \"kubernetes.io/projected/7ab060b7-ab03-4bd9-b917-9a4945a0b614-kube-api-access-zwtx8\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.719895 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719798    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.719895 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719817    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.719895 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719836    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.719895 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719878    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.720086 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719902    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.720086 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.719929    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-web-config\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821004 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.820975    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821125 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821013    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821125 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821041    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7ab060b7-ab03-4bd9-b917-9a4945a0b614-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821125 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821072    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821125 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821094    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821125 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821122    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821359 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821151    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ab060b7-ab03-4bd9-b917-9a4945a0b614-config-out\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821359 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821178    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821359 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821200    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwtx8\" (UniqueName: \"kubernetes.io/projected/7ab060b7-ab03-4bd9-b917-9a4945a0b614-kube-api-access-zwtx8\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821359 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821232    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821359 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821263    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821359 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821291    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821359 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821318    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821359 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821338    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821775 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821369    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-web-config\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821775 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821407    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ab060b7-ab03-4bd9-b917-9a4945a0b614-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821775 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821437    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821775 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821465    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821775 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821493    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-config\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821775 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821527    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.821775 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821539    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7ab060b7-ab03-4bd9-b917-9a4945a0b614-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.822150 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.821811    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.824596 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.824001    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ab060b7-ab03-4bd9-b917-9a4945a0b614-config-out\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.824596 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.824275    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.824596 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.824459    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ab060b7-ab03-4bd9-b917-9a4945a0b614-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.824596 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.824481    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.824596 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.824492    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.824596 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.824502    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.824596 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.824508    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.826054 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.825202    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.826054 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.825336    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.826054 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.825525    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.826054 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.825775    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.826054 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.825808    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.826054 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.826028    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ab060b7-ab03-4bd9-b917-9a4945a0b614-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.827046 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.827014    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-config\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.827151 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.827083    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.827496 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.827475    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.827773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.827755    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ab060b7-ab03-4bd9-b917-9a4945a0b614-web-config\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.845308 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.845290    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwtx8\" (UniqueName: \"kubernetes.io/projected/7ab060b7-ab03-4bd9-b917-9a4945a0b614-kube-api-access-zwtx8\") pod \"prometheus-k8s-0\" (UID: \"7ab060b7-ab03-4bd9-b917-9a4945a0b614\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:42.914331 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:42.914311    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:35:43.237388 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:43.237302    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:35:43.241127 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:35:43.241099    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab060b7_ab03_4bd9_b917_9a4945a0b614.slice/crio-16862044a32c6b9ed083b838e524d0286bb286fb3f43c996731cb61cf8fd6dbd WatchSource:0}: Error finding container 16862044a32c6b9ed083b838e524d0286bb286fb3f43c996731cb61cf8fd6dbd: Status 404 returned error can't find the container with id 16862044a32c6b9ed083b838e524d0286bb286fb3f43c996731cb61cf8fd6dbd
Oct 30 00:35:43.504025 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:43.503950    6099 generic.go:356] "Generic (PLEG): container finished" podID="7ab060b7-ab03-4bd9-b917-9a4945a0b614" containerID="c550ab4e857f3d1eba8b9538f0480942e4b58940dd6fa4762437df2086ed42ec" exitCode=0
Oct 30 00:35:43.504159 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:43.504023    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ab060b7-ab03-4bd9-b917-9a4945a0b614","Type":"ContainerDied","Data":"c550ab4e857f3d1eba8b9538f0480942e4b58940dd6fa4762437df2086ed42ec"}
Oct 30 00:35:43.504159 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:43.504086    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ab060b7-ab03-4bd9-b917-9a4945a0b614","Type":"ContainerStarted","Data":"16862044a32c6b9ed083b838e524d0286bb286fb3f43c996731cb61cf8fd6dbd"}
Oct 30 00:35:44.258839 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:44.258804    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="026218e5-d0ac-48b3-9023-f245d5cf9848" path="/var/lib/kubelet/pods/026218e5-d0ac-48b3-9023-f245d5cf9848/volumes"
Oct 30 00:35:44.511658 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:44.511581    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ab060b7-ab03-4bd9-b917-9a4945a0b614","Type":"ContainerStarted","Data":"53e21da24bf87ca8e34ebfef2dafbfc132bd1ae6a796451b0c53902b4d959dc3"}
Oct 30 00:35:44.511810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:44.511658    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ab060b7-ab03-4bd9-b917-9a4945a0b614","Type":"ContainerStarted","Data":"992b3d9401559a56adc1ef262682f3989d90155e608137ec386fe6084b9b2cd8"}
Oct 30 00:35:44.511810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:44.511699    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ab060b7-ab03-4bd9-b917-9a4945a0b614","Type":"ContainerStarted","Data":"c6f5cc2c38ab8126414aaa3da3bfa06e69bdbc9347a8d62448abaff3c7f579a7"}
Oct 30 00:35:44.511810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:44.511759    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ab060b7-ab03-4bd9-b917-9a4945a0b614","Type":"ContainerStarted","Data":"50926b1ee77c0b21dd639c2ed504fd81d0d839c029004e48a46d55a11888dabb"}
Oct 30 00:35:44.511810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:44.511794    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ab060b7-ab03-4bd9-b917-9a4945a0b614","Type":"ContainerStarted","Data":"63de5354e6988b9d8f3f87219cd46e5b0a0bb07f32429dfa888ac7e173bb9dcf"}
Oct 30 00:35:44.511956 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:44.511828    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ab060b7-ab03-4bd9-b917-9a4945a0b614","Type":"ContainerStarted","Data":"06601f57f8edc0c2b1e3cab965f391805a1f9ded116b77a23751e312871f970e"}
Oct 30 00:35:44.572533 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:44.572485    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.558277854 podStartE2EDuration="2.572471511s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.193657ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:44.571491291 +0000 UTC m=+38.428399048" watchObservedRunningTime="2026-10-30 00:35:44.572471511 +0000 UTC m=+38.429379265"
Oct 30 00:35:46.627023 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:35:46.626986    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:35:47.914752 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:35:47.914700    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:36:16.627792 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:36:16.627752    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:36:24.037865 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:36:24.037817    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:36:24.053289 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:36:24.053266    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:36:24.697921 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:36:24.697896    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:36:46.627549 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:36:46.627507    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:37:16.628326 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:37:16.628278    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:37:46.628192 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:37:46.628152    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:38:01.489139 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:01.489108    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-z4zgq"]
Oct 30 00:38:01.491796 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:01.491764    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-z4zgq"
Oct 30 00:38:01.504218 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:01.504201    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 00:38:01.514229 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:01.514204    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 00:38:01.514890 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:01.514868    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-z4zgq"]
Oct 30 00:38:01.543554 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:01.543529    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-djdcg\""
Oct 30 00:38:01.556932 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:01.556912    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0e1cb5b-9ddd-46b8-a1d8-4113d9218580-bound-sa-token\") pod \"cert-manager-597b95cb84-z4zgq\" (UID: \"d0e1cb5b-9ddd-46b8-a1d8-4113d9218580\") " pod="cert-manager/cert-manager-597b95cb84-z4zgq"
Oct 30 00:38:01.557001 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:01.556937    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhphl\" (UniqueName: \"kubernetes.io/projected/d0e1cb5b-9ddd-46b8-a1d8-4113d9218580-kube-api-access-fhphl\") pod \"cert-manager-597b95cb84-z4zgq\" (UID: \"d0e1cb5b-9ddd-46b8-a1d8-4113d9218580\") " pod="cert-manager/cert-manager-597b95cb84-z4zgq"
Oct 30 00:38:01.658101 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:01.658068    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0e1cb5b-9ddd-46b8-a1d8-4113d9218580-bound-sa-token\") pod \"cert-manager-597b95cb84-z4zgq\" (UID: \"d0e1cb5b-9ddd-46b8-a1d8-4113d9218580\") " pod="cert-manager/cert-manager-597b95cb84-z4zgq"
Oct 30 00:38:01.658101 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:01.658100    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhphl\" (UniqueName: \"kubernetes.io/projected/d0e1cb5b-9ddd-46b8-a1d8-4113d9218580-kube-api-access-fhphl\") pod \"cert-manager-597b95cb84-z4zgq\" (UID: \"d0e1cb5b-9ddd-46b8-a1d8-4113d9218580\") " pod="cert-manager/cert-manager-597b95cb84-z4zgq"
Oct 30 00:38:01.679475 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:01.679447    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0e1cb5b-9ddd-46b8-a1d8-4113d9218580-bound-sa-token\") pod \"cert-manager-597b95cb84-z4zgq\" (UID: \"d0e1cb5b-9ddd-46b8-a1d8-4113d9218580\") " pod="cert-manager/cert-manager-597b95cb84-z4zgq"
Oct 30 00:38:01.679590 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:01.679486    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhphl\" (UniqueName: \"kubernetes.io/projected/d0e1cb5b-9ddd-46b8-a1d8-4113d9218580-kube-api-access-fhphl\") pod \"cert-manager-597b95cb84-z4zgq\" (UID: \"d0e1cb5b-9ddd-46b8-a1d8-4113d9218580\") " pod="cert-manager/cert-manager-597b95cb84-z4zgq"
Oct 30 00:38:01.812075 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:01.812044    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-z4zgq"
Oct 30 00:38:02.103121 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:02.103045    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-z4zgq"]
Oct 30 00:38:02.107094 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:38:02.107064    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0e1cb5b_9ddd_46b8_a1d8_4113d9218580.slice/crio-e8da702d90a6f487745c6e60b008d39daad0a7af5fbc6b88884602ddabf71047 WatchSource:0}: Error finding container e8da702d90a6f487745c6e60b008d39daad0a7af5fbc6b88884602ddabf71047: Status 404 returned error can't find the container with id e8da702d90a6f487745c6e60b008d39daad0a7af5fbc6b88884602ddabf71047
Oct 30 00:38:02.108937 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:02.108918    6099 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:38:03.040600 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:03.040557    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-z4zgq" event={"ID":"d0e1cb5b-9ddd-46b8-a1d8-4113d9218580","Type":"ContainerStarted","Data":"e8da702d90a6f487745c6e60b008d39daad0a7af5fbc6b88884602ddabf71047"}
Oct 30 00:38:06.054770 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:06.054734    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-z4zgq" event={"ID":"d0e1cb5b-9ddd-46b8-a1d8-4113d9218580","Type":"ContainerStarted","Data":"1c3db3ca1bc2da6bd482b5550fbdd1ca2f111231857726619f0bd7dd2e64dfd0"}
Oct 30 00:38:06.090716 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:06.090657    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-z4zgq" podStartSLOduration=1.9840100729999999 podStartE2EDuration="5.090640176s" totalImagesPullingTime="3.106630103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:06.089961868 +0000 UTC m=+179.946869621" watchObservedRunningTime="2026-10-30 00:38:06.090640176 +0000 UTC m=+179.947547928"
Oct 30 00:38:16.628201 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:38:16.628166    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:38:25.934985 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:25.934908    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"]
Oct 30 00:38:25.938312 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:25.938290    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"
Oct 30 00:38:25.957272 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:25.957252    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 00:38:25.965836 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:25.965807    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7cc9f2f2-bf4f-47e7-9c78-96851128d898-manager-config\") pod \"lws-controller-manager-7fdd95d5df-g4hnz\" (UID: \"7cc9f2f2-bf4f-47e7-9c78-96851128d898\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"
Oct 30 00:38:25.965943 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:25.965858    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cc9f2f2-bf4f-47e7-9c78-96851128d898-cert\") pod \"lws-controller-manager-7fdd95d5df-g4hnz\" (UID: \"7cc9f2f2-bf4f-47e7-9c78-96851128d898\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"
Oct 30 00:38:25.966012 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:25.965992    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7cc9f2f2-bf4f-47e7-9c78-96851128d898-metrics-cert\") pod \"lws-controller-manager-7fdd95d5df-g4hnz\" (UID: \"7cc9f2f2-bf4f-47e7-9c78-96851128d898\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"
Oct 30 00:38:25.966075 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:25.966054    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4s6n\" (UniqueName: \"kubernetes.io/projected/7cc9f2f2-bf4f-47e7-9c78-96851128d898-kube-api-access-k4s6n\") pod \"lws-controller-manager-7fdd95d5df-g4hnz\" (UID: \"7cc9f2f2-bf4f-47e7-9c78-96851128d898\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"
Oct 30 00:38:25.967406 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:25.967382    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 00:38:25.967406 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:25.967401    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:38:25.967545 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:25.967428    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 00:38:25.968073 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:25.968055    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"]
Oct 30 00:38:26.059536 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:26.059506    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2sxm9\""
Oct 30 00:38:26.059536 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:26.059507    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 00:38:26.067043 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:26.067013    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4s6n\" (UniqueName: \"kubernetes.io/projected/7cc9f2f2-bf4f-47e7-9c78-96851128d898-kube-api-access-k4s6n\") pod \"lws-controller-manager-7fdd95d5df-g4hnz\" (UID: \"7cc9f2f2-bf4f-47e7-9c78-96851128d898\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"
Oct 30 00:38:26.067219 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:26.067121    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7cc9f2f2-bf4f-47e7-9c78-96851128d898-manager-config\") pod \"lws-controller-manager-7fdd95d5df-g4hnz\" (UID: \"7cc9f2f2-bf4f-47e7-9c78-96851128d898\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"
Oct 30 00:38:26.067219 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:26.067154    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cc9f2f2-bf4f-47e7-9c78-96851128d898-cert\") pod \"lws-controller-manager-7fdd95d5df-g4hnz\" (UID: \"7cc9f2f2-bf4f-47e7-9c78-96851128d898\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"
Oct 30 00:38:26.067219 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:26.067191    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7cc9f2f2-bf4f-47e7-9c78-96851128d898-metrics-cert\") pod \"lws-controller-manager-7fdd95d5df-g4hnz\" (UID: \"7cc9f2f2-bf4f-47e7-9c78-96851128d898\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"
Oct 30 00:38:26.067932 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:26.067892    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7cc9f2f2-bf4f-47e7-9c78-96851128d898-manager-config\") pod \"lws-controller-manager-7fdd95d5df-g4hnz\" (UID: \"7cc9f2f2-bf4f-47e7-9c78-96851128d898\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"
Oct 30 00:38:26.069903 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:26.069873    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7cc9f2f2-bf4f-47e7-9c78-96851128d898-metrics-cert\") pod \"lws-controller-manager-7fdd95d5df-g4hnz\" (UID: \"7cc9f2f2-bf4f-47e7-9c78-96851128d898\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"
Oct 30 00:38:26.069903 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:26.069888    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cc9f2f2-bf4f-47e7-9c78-96851128d898-cert\") pod \"lws-controller-manager-7fdd95d5df-g4hnz\" (UID: \"7cc9f2f2-bf4f-47e7-9c78-96851128d898\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"
Oct 30 00:38:26.089304 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:26.089274    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4s6n\" (UniqueName: \"kubernetes.io/projected/7cc9f2f2-bf4f-47e7-9c78-96851128d898-kube-api-access-k4s6n\") pod \"lws-controller-manager-7fdd95d5df-g4hnz\" (UID: \"7cc9f2f2-bf4f-47e7-9c78-96851128d898\") " pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"
Oct 30 00:38:26.249746 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:26.249648    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"
Oct 30 00:38:26.546293 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:26.546261    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"]
Oct 30 00:38:26.549501 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:38:26.549474    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc9f2f2_bf4f_47e7_9c78_96851128d898.slice/crio-9199d71033a0f8b299c6f81b243634b44145506fc733fd85df1a044299a89ab5 WatchSource:0}: Error finding container 9199d71033a0f8b299c6f81b243634b44145506fc733fd85df1a044299a89ab5: Status 404 returned error can't find the container with id 9199d71033a0f8b299c6f81b243634b44145506fc733fd85df1a044299a89ab5
Oct 30 00:38:27.139506 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:27.139475    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz" event={"ID":"7cc9f2f2-bf4f-47e7-9c78-96851128d898","Type":"ContainerStarted","Data":"9199d71033a0f8b299c6f81b243634b44145506fc733fd85df1a044299a89ab5"}
Oct 30 00:38:28.305368 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.305336    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"]
Oct 30 00:38:28.314516 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.314494    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.348541 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.348510    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"]
Oct 30 00:38:28.388114 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.388083    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fade35d-c453-4e88-a630-605cf4a7a364-tmp\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-k29rq\" (UID: \"0fade35d-c453-4e88-a630-605cf4a7a364\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.388346 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.388284    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0fade35d-c453-4e88-a630-605cf4a7a364-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-k29rq\" (UID: \"0fade35d-c453-4e88-a630-605cf4a7a364\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.388428 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.388369    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0fade35d-c453-4e88-a630-605cf4a7a364-manifests\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-k29rq\" (UID: \"0fade35d-c453-4e88-a630-605cf4a7a364\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.388428 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.388411    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0fade35d-c453-4e88-a630-605cf4a7a364-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-k29rq\" (UID: \"0fade35d-c453-4e88-a630-605cf4a7a364\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.388549 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.388496    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fade35d-c453-4e88-a630-605cf4a7a364-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-k29rq\" (UID: \"0fade35d-c453-4e88-a630-605cf4a7a364\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.388549 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.388529    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68wsd\" (UniqueName: \"kubernetes.io/projected/0fade35d-c453-4e88-a630-605cf4a7a364-kube-api-access-68wsd\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-k29rq\" (UID: \"0fade35d-c453-4e88-a630-605cf4a7a364\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.406422 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.406397    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 00:38:28.406694 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.406677    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 00:38:28.406804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.406754    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-725k5\""
Oct 30 00:38:28.406804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.406783    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 00:38:28.406804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.406799    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 00:38:28.406936 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.406807    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 00:38:28.489524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.489487    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fade35d-c453-4e88-a630-605cf4a7a364-tmp\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-k29rq\" (UID: \"0fade35d-c453-4e88-a630-605cf4a7a364\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.489722 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.489561    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0fade35d-c453-4e88-a630-605cf4a7a364-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-k29rq\" (UID: \"0fade35d-c453-4e88-a630-605cf4a7a364\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.489722 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.489596    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0fade35d-c453-4e88-a630-605cf4a7a364-manifests\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-k29rq\" (UID: \"0fade35d-c453-4e88-a630-605cf4a7a364\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.489722 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.489618    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0fade35d-c453-4e88-a630-605cf4a7a364-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-k29rq\" (UID: \"0fade35d-c453-4e88-a630-605cf4a7a364\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.489722 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.489650    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fade35d-c453-4e88-a630-605cf4a7a364-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-k29rq\" (UID: \"0fade35d-c453-4e88-a630-605cf4a7a364\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.489722 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.489672    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68wsd\" (UniqueName: \"kubernetes.io/projected/0fade35d-c453-4e88-a630-605cf4a7a364-kube-api-access-68wsd\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-k29rq\" (UID: \"0fade35d-c453-4e88-a630-605cf4a7a364\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.490060 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.490038    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0fade35d-c453-4e88-a630-605cf4a7a364-manifests\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-k29rq\" (UID: \"0fade35d-c453-4e88-a630-605cf4a7a364\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.492210 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.492185    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fade35d-c453-4e88-a630-605cf4a7a364-tmp\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-k29rq\" (UID: \"0fade35d-c453-4e88-a630-605cf4a7a364\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.492777 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.492749    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0fade35d-c453-4e88-a630-605cf4a7a364-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-k29rq\" (UID: \"0fade35d-c453-4e88-a630-605cf4a7a364\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.494089 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.494068    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fade35d-c453-4e88-a630-605cf4a7a364-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-k29rq\" (UID: \"0fade35d-c453-4e88-a630-605cf4a7a364\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.494264 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.494245    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0fade35d-c453-4e88-a630-605cf4a7a364-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-k29rq\" (UID: \"0fade35d-c453-4e88-a630-605cf4a7a364\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.514055 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.514034    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68wsd\" (UniqueName: \"kubernetes.io/projected/0fade35d-c453-4e88-a630-605cf4a7a364-kube-api-access-68wsd\") pod \"opendatahub-operator-controller-manager-6f85c8f7c-k29rq\" (UID: \"0fade35d-c453-4e88-a630-605cf4a7a364\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.632678 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.632646    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:28.985870 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:28.985843    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"]
Oct 30 00:38:28.989792 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:38:28.989761    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fade35d_c453_4e88_a630_605cf4a7a364.slice/crio-c7e312862e94d623853f0cb9542194e268bd80b76cf516798d17313bd2e18bcc WatchSource:0}: Error finding container c7e312862e94d623853f0cb9542194e268bd80b76cf516798d17313bd2e18bcc: Status 404 returned error can't find the container with id c7e312862e94d623853f0cb9542194e268bd80b76cf516798d17313bd2e18bcc
Oct 30 00:38:29.149286 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:29.149248    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq" event={"ID":"0fade35d-c453-4e88-a630-605cf4a7a364","Type":"ContainerStarted","Data":"c7e312862e94d623853f0cb9542194e268bd80b76cf516798d17313bd2e18bcc"}
Oct 30 00:38:29.150689 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:29.150661    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz" event={"ID":"7cc9f2f2-bf4f-47e7-9c78-96851128d898","Type":"ContainerStarted","Data":"27e6314bb98012df760163a5bb46b606f1ce3c6376070cc71e841a3158c6abc3"}
Oct 30 00:38:29.150866 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:29.150773    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"
Oct 30 00:38:29.188563 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:29.188515    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz" podStartSLOduration=1.779551052 podStartE2EDuration="4.188504345s" totalImagesPullingTime="2.408953293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:29.187959227 +0000 UTC m=+203.044866983" watchObservedRunningTime="2026-10-30 00:38:29.188504345 +0000 UTC m=+203.045412093"
Oct 30 00:38:32.167058 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:32.166979    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq" event={"ID":"0fade35d-c453-4e88-a630-605cf4a7a364","Type":"ContainerStarted","Data":"cea40cf02d5290d09b63fd4b91ca54ccc54bd20b938f815abac0d125d73bc99e"}
Oct 30 00:38:33.172100 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:33.172068    6099 generic.go:356] "Generic (PLEG): container finished" podID="0fade35d-c453-4e88-a630-605cf4a7a364" containerID="cea40cf02d5290d09b63fd4b91ca54ccc54bd20b938f815abac0d125d73bc99e" exitCode=0
Oct 30 00:38:33.172484 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:33.172158    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq" event={"ID":"0fade35d-c453-4e88-a630-605cf4a7a364","Type":"ContainerDied","Data":"cea40cf02d5290d09b63fd4b91ca54ccc54bd20b938f815abac0d125d73bc99e"}
Oct 30 00:38:34.177989 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:34.177950    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq" event={"ID":"0fade35d-c453-4e88-a630-605cf4a7a364","Type":"ContainerStarted","Data":"89640330b96a620ff0cde691c98644a5ee564308fabb79f288fce4e3a76e1676"}
Oct 30 00:38:34.178466 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:34.178115    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:34.255497 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:34.255439    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq" podStartSLOduration=2.569195789 podStartE2EDuration="6.255425958s" totalImagesPullingTime="3.27305223s" totalInitContainerRuntime="413.177939ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:34.245451939 +0000 UTC m=+208.102359691" watchObservedRunningTime="2026-10-30 00:38:34.255425958 +0000 UTC m=+208.112333712"
Oct 30 00:38:40.158780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:40.158749    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7fdd95d5df-g4hnz"
Oct 30 00:38:45.184226 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:45.184191    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq"
Oct 30 00:38:46.627408 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:38:46.627368    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:38:47.996821 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:47.996785    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-64d5b5b7d7-nttd5"]
Oct 30 00:38:47.999202 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:47.999177    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-nttd5"
Oct 30 00:38:48.012409 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:48.012386    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 00:38:48.012809 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:48.012790    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 00:38:48.022583 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:48.022557    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 00:38:48.022740 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:48.022556    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sk75l\""
Oct 30 00:38:48.022740 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:48.022671    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 00:38:48.023636 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:48.023612    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64d5b5b7d7-nttd5"]
Oct 30 00:38:48.055846 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:48.055820    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270-tls-certs\") pod \"kube-auth-proxy-64d5b5b7d7-nttd5\" (UID: \"3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270\") " pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-nttd5"
Oct 30 00:38:48.055967 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:48.055870    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270-tmp\") pod \"kube-auth-proxy-64d5b5b7d7-nttd5\" (UID: \"3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270\") " pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-nttd5"
Oct 30 00:38:48.055967 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:48.055927    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dxtw\" (UniqueName: \"kubernetes.io/projected/3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270-kube-api-access-7dxtw\") pod \"kube-auth-proxy-64d5b5b7d7-nttd5\" (UID: \"3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270\") " pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-nttd5"
Oct 30 00:38:48.156397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:48.156368    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270-tls-certs\") pod \"kube-auth-proxy-64d5b5b7d7-nttd5\" (UID: \"3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270\") " pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-nttd5"
Oct 30 00:38:48.156523 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:48.156418    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270-tmp\") pod \"kube-auth-proxy-64d5b5b7d7-nttd5\" (UID: \"3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270\") " pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-nttd5"
Oct 30 00:38:48.156563 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:48.156540    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dxtw\" (UniqueName: \"kubernetes.io/projected/3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270-kube-api-access-7dxtw\") pod \"kube-auth-proxy-64d5b5b7d7-nttd5\" (UID: \"3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270\") " pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-nttd5"
Oct 30 00:38:48.158606 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:48.158576    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270-tmp\") pod \"kube-auth-proxy-64d5b5b7d7-nttd5\" (UID: \"3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270\") " pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-nttd5"
Oct 30 00:38:48.158822 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:48.158805    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270-tls-certs\") pod \"kube-auth-proxy-64d5b5b7d7-nttd5\" (UID: \"3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270\") " pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-nttd5"
Oct 30 00:38:48.179636 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:48.179611    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dxtw\" (UniqueName: \"kubernetes.io/projected/3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270-kube-api-access-7dxtw\") pod \"kube-auth-proxy-64d5b5b7d7-nttd5\" (UID: \"3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270\") " pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-nttd5"
Oct 30 00:38:48.309801 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:48.309779    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-nttd5"
Oct 30 00:38:48.603527 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:48.603445    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64d5b5b7d7-nttd5"]
Oct 30 00:38:48.606563 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:38:48.606530    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5daaf9_d4a9_4d5c_bb8f_b3ea12c55270.slice/crio-e37b4d67d688466c01ab714710930e40e701e7300847c343bb26383fe80e1810 WatchSource:0}: Error finding container e37b4d67d688466c01ab714710930e40e701e7300847c343bb26383fe80e1810: Status 404 returned error can't find the container with id e37b4d67d688466c01ab714710930e40e701e7300847c343bb26383fe80e1810
Oct 30 00:38:49.242122 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:49.242082    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-nttd5" event={"ID":"3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270","Type":"ContainerStarted","Data":"e37b4d67d688466c01ab714710930e40e701e7300847c343bb26383fe80e1810"}
Oct 30 00:38:51.252374 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:51.252283    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-nttd5" event={"ID":"3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270","Type":"ContainerStarted","Data":"3dcffd04984b25d3142d8e72de4d923623e852a7368d931214ccbf00a7e369de"}
Oct 30 00:38:51.288038 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:38:51.287992    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-64d5b5b7d7-nttd5" podStartSLOduration=2.000825529 podStartE2EDuration="4.287977841s" totalImagesPullingTime="2.287152312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:51.287510839 +0000 UTC m=+225.144418593" watchObservedRunningTime="2026-10-30 00:38:51.287977841 +0000 UTC m=+225.144885596"
Oct 30 00:39:16.627739 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:39:16.627685    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:39:46.627719 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:39:46.627667    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:39:52.005649 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:52.005619    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-lbcfw"]
Oct 30 00:39:52.008888 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:52.008854    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-lbcfw"
Oct 30 00:39:52.028538 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:52.028511    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-lbcfw"]
Oct 30 00:39:52.037743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:52.037715    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 00:39:52.037743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:52.037742    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4nxxh\""
Oct 30 00:39:52.037904 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:52.037781    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 00:39:52.108554 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:52.108523    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6spzv\" (UniqueName: \"kubernetes.io/projected/938fce0b-97c1-4d08-9565-f92cd7e064c7-kube-api-access-6spzv\") pod \"odh-model-controller-f87fcf56c-lbcfw\" (UID: \"938fce0b-97c1-4d08-9565-f92cd7e064c7\") " pod="opendatahub/odh-model-controller-f87fcf56c-lbcfw"
Oct 30 00:39:52.108728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:52.108688    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/938fce0b-97c1-4d08-9565-f92cd7e064c7-metrics-certs\") pod \"odh-model-controller-f87fcf56c-lbcfw\" (UID: \"938fce0b-97c1-4d08-9565-f92cd7e064c7\") " pod="opendatahub/odh-model-controller-f87fcf56c-lbcfw"
Oct 30 00:39:52.108796 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:52.108739    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/938fce0b-97c1-4d08-9565-f92cd7e064c7-cert\") pod \"odh-model-controller-f87fcf56c-lbcfw\" (UID: \"938fce0b-97c1-4d08-9565-f92cd7e064c7\") " pod="opendatahub/odh-model-controller-f87fcf56c-lbcfw"
Oct 30 00:39:52.209773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:52.209742    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/938fce0b-97c1-4d08-9565-f92cd7e064c7-cert\") pod \"odh-model-controller-f87fcf56c-lbcfw\" (UID: \"938fce0b-97c1-4d08-9565-f92cd7e064c7\") " pod="opendatahub/odh-model-controller-f87fcf56c-lbcfw"
Oct 30 00:39:52.209920 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:52.209780    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6spzv\" (UniqueName: \"kubernetes.io/projected/938fce0b-97c1-4d08-9565-f92cd7e064c7-kube-api-access-6spzv\") pod \"odh-model-controller-f87fcf56c-lbcfw\" (UID: \"938fce0b-97c1-4d08-9565-f92cd7e064c7\") " pod="opendatahub/odh-model-controller-f87fcf56c-lbcfw"
Oct 30 00:39:52.209920 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:52.209856    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/938fce0b-97c1-4d08-9565-f92cd7e064c7-metrics-certs\") pod \"odh-model-controller-f87fcf56c-lbcfw\" (UID: \"938fce0b-97c1-4d08-9565-f92cd7e064c7\") " pod="opendatahub/odh-model-controller-f87fcf56c-lbcfw"
Oct 30 00:39:52.213128 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:52.213102    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/938fce0b-97c1-4d08-9565-f92cd7e064c7-metrics-certs\") pod \"odh-model-controller-f87fcf56c-lbcfw\" (UID: \"938fce0b-97c1-4d08-9565-f92cd7e064c7\") " pod="opendatahub/odh-model-controller-f87fcf56c-lbcfw"
Oct 30 00:39:52.213241 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:52.213104    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/938fce0b-97c1-4d08-9565-f92cd7e064c7-cert\") pod \"odh-model-controller-f87fcf56c-lbcfw\" (UID: \"938fce0b-97c1-4d08-9565-f92cd7e064c7\") " pod="opendatahub/odh-model-controller-f87fcf56c-lbcfw"
Oct 30 00:39:52.237175 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:52.237142    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6spzv\" (UniqueName: \"kubernetes.io/projected/938fce0b-97c1-4d08-9565-f92cd7e064c7-kube-api-access-6spzv\") pod \"odh-model-controller-f87fcf56c-lbcfw\" (UID: \"938fce0b-97c1-4d08-9565-f92cd7e064c7\") " pod="opendatahub/odh-model-controller-f87fcf56c-lbcfw"
Oct 30 00:39:52.322189 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:52.322159    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-lbcfw"
Oct 30 00:39:52.613836 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:52.613691    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-lbcfw"]
Oct 30 00:39:52.618079 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:39:52.618053    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938fce0b_97c1_4d08_9565_f92cd7e064c7.slice/crio-6924c74f2e6a042ae6de8ec53eac130221306c61fadafd32e23bff0a20942459 WatchSource:0}: Error finding container 6924c74f2e6a042ae6de8ec53eac130221306c61fadafd32e23bff0a20942459: Status 404 returned error can't find the container with id 6924c74f2e6a042ae6de8ec53eac130221306c61fadafd32e23bff0a20942459
Oct 30 00:39:53.502144 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:53.502103    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-lbcfw" event={"ID":"938fce0b-97c1-4d08-9565-f92cd7e064c7","Type":"ContainerStarted","Data":"6924c74f2e6a042ae6de8ec53eac130221306c61fadafd32e23bff0a20942459"}
Oct 30 00:39:55.512666 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:55.512577    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-lbcfw" event={"ID":"938fce0b-97c1-4d08-9565-f92cd7e064c7","Type":"ContainerStarted","Data":"50e56370d221dc9ac79a27a6e535d623ee408a2304bd9eb3c8e01733662b57e3"}
Oct 30 00:39:55.512666 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:55.512646    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-lbcfw"
Oct 30 00:39:55.551014 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:39:55.550957    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-lbcfw" podStartSLOduration=2.076731266 podStartE2EDuration="4.550940713s" totalImagesPullingTime="2.474209447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:55.550281287 +0000 UTC m=+289.407189039" watchObservedRunningTime="2026-10-30 00:39:55.550940713 +0000 UTC m=+289.407848465"
Oct 30 00:40:06.290643 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:40:06.290616    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 00:40:06.293760 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:40:06.293741    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 00:40:06.519484 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:40:06.519459    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-lbcfw"
Oct 30 00:40:16.627366 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:40:16.627327    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:40:46.628603 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:40:46.628557    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:41:06.485960 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:06.485929    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qnjq"]
Oct 30 00:41:06.489749 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:06.489727    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qnjq"
Oct 30 00:41:06.502158 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:06.502138    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 00:41:06.512171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:06.512149    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 00:41:06.512271 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:06.512184    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 00:41:06.512369 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:06.512349    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 00:41:06.512482 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:06.512350    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qsgkj\""
Oct 30 00:41:06.513525 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:06.513500    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qnjq"]
Oct 30 00:41:06.587845 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:06.587817    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/85eff882-1ab7-4def-92d7-47c8135c44c9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8qnjq\" (UID: \"85eff882-1ab7-4def-92d7-47c8135c44c9\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qnjq"
Oct 30 00:41:06.587969 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:06.587883    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85eff882-1ab7-4def-92d7-47c8135c44c9-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8qnjq\" (UID: \"85eff882-1ab7-4def-92d7-47c8135c44c9\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qnjq"
Oct 30 00:41:06.588018 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:06.587995    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcj2h\" (UniqueName: \"kubernetes.io/projected/85eff882-1ab7-4def-92d7-47c8135c44c9-kube-api-access-wcj2h\") pod \"kuadrant-console-plugin-66c4fb575d-8qnjq\" (UID: \"85eff882-1ab7-4def-92d7-47c8135c44c9\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qnjq"
Oct 30 00:41:06.689381 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:06.689358    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85eff882-1ab7-4def-92d7-47c8135c44c9-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8qnjq\" (UID: \"85eff882-1ab7-4def-92d7-47c8135c44c9\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qnjq"
Oct 30 00:41:06.689490 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:06.689408    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcj2h\" (UniqueName: \"kubernetes.io/projected/85eff882-1ab7-4def-92d7-47c8135c44c9-kube-api-access-wcj2h\") pod \"kuadrant-console-plugin-66c4fb575d-8qnjq\" (UID: \"85eff882-1ab7-4def-92d7-47c8135c44c9\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qnjq"
Oct 30 00:41:06.689490 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:06.689433    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/85eff882-1ab7-4def-92d7-47c8135c44c9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8qnjq\" (UID: \"85eff882-1ab7-4def-92d7-47c8135c44c9\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qnjq"
Oct 30 00:41:06.690033 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:06.690017    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85eff882-1ab7-4def-92d7-47c8135c44c9-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8qnjq\" (UID: \"85eff882-1ab7-4def-92d7-47c8135c44c9\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qnjq"
Oct 30 00:41:06.692008 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:06.691984    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/85eff882-1ab7-4def-92d7-47c8135c44c9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8qnjq\" (UID: \"85eff882-1ab7-4def-92d7-47c8135c44c9\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qnjq"
Oct 30 00:41:06.710162 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:06.710135    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcj2h\" (UniqueName: \"kubernetes.io/projected/85eff882-1ab7-4def-92d7-47c8135c44c9-kube-api-access-wcj2h\") pod \"kuadrant-console-plugin-66c4fb575d-8qnjq\" (UID: \"85eff882-1ab7-4def-92d7-47c8135c44c9\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qnjq"
Oct 30 00:41:06.802212 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:06.802181    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qnjq"
Oct 30 00:41:07.090760 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:07.090663    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qnjq"]
Oct 30 00:41:07.095178 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:41:07.095140    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85eff882_1ab7_4def_92d7_47c8135c44c9.slice/crio-3fb15deafdeb312676596ee509409efdf759d3b32a14c47c637f2dede7e7068b WatchSource:0}: Error finding container 3fb15deafdeb312676596ee509409efdf759d3b32a14c47c637f2dede7e7068b: Status 404 returned error can't find the container with id 3fb15deafdeb312676596ee509409efdf759d3b32a14c47c637f2dede7e7068b
Oct 30 00:41:07.814756 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:07.814721    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qnjq" event={"ID":"85eff882-1ab7-4def-92d7-47c8135c44c9","Type":"ContainerStarted","Data":"3fb15deafdeb312676596ee509409efdf759d3b32a14c47c637f2dede7e7068b"}
Oct 30 00:41:12.838825 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:12.838781    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qnjq" event={"ID":"85eff882-1ab7-4def-92d7-47c8135c44c9","Type":"ContainerStarted","Data":"313d99b4563f65d30fb7d57aecc1cb2a09db820a6e27d267ad1d662acf6f1d1b"}
Oct 30 00:41:12.873656 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:12.873614    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qnjq" podStartSLOduration=1.871115288 podStartE2EDuration="6.873602485s" totalImagesPullingTime="5.002487197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:12.873223031 +0000 UTC m=+366.730130786" watchObservedRunningTime="2026-10-30 00:41:12.873602485 +0000 UTC m=+366.730510237"
Oct 30 00:41:16.264320 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:16.264250    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p"]
Oct 30 00:41:16.268085 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:16.268068    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p"
Oct 30 00:41:16.296222 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:16.296194    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ac3f48f-f59a-42bc-9cf9-f4662ea24771-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-t6v5p\" (UID: \"8ac3f48f-f59a-42bc-9cf9-f4662ea24771\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p"
Oct 30 00:41:16.296363 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:16.296245    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t24x\" (UniqueName: \"kubernetes.io/projected/8ac3f48f-f59a-42bc-9cf9-f4662ea24771-kube-api-access-6t24x\") pod \"kuadrant-operator-controller-manager-6b76784d48-t6v5p\" (UID: \"8ac3f48f-f59a-42bc-9cf9-f4662ea24771\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p"
Oct 30 00:41:16.302268 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:16.302245    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-87vj2\""
Oct 30 00:41:16.303735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:16.303678    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p"]
Oct 30 00:41:16.397631 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:16.397602    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ac3f48f-f59a-42bc-9cf9-f4662ea24771-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-t6v5p\" (UID: \"8ac3f48f-f59a-42bc-9cf9-f4662ea24771\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p"
Oct 30 00:41:16.397787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:16.397653    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t24x\" (UniqueName: \"kubernetes.io/projected/8ac3f48f-f59a-42bc-9cf9-f4662ea24771-kube-api-access-6t24x\") pod \"kuadrant-operator-controller-manager-6b76784d48-t6v5p\" (UID: \"8ac3f48f-f59a-42bc-9cf9-f4662ea24771\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p"
Oct 30 00:41:16.398030 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:16.398009    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ac3f48f-f59a-42bc-9cf9-f4662ea24771-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-t6v5p\" (UID: \"8ac3f48f-f59a-42bc-9cf9-f4662ea24771\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p"
Oct 30 00:41:16.418656 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:16.418631    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t24x\" (UniqueName: \"kubernetes.io/projected/8ac3f48f-f59a-42bc-9cf9-f4662ea24771-kube-api-access-6t24x\") pod \"kuadrant-operator-controller-manager-6b76784d48-t6v5p\" (UID: \"8ac3f48f-f59a-42bc-9cf9-f4662ea24771\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p"
Oct 30 00:41:16.467629 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:16.467604    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p"]
Oct 30 00:41:16.467865 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:16.467853    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p"
Oct 30 00:41:16.627349 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:41:16.627312    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:41:16.786663 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:16.786584    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p"]
Oct 30 00:41:16.789916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:41:16.789890    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac3f48f_f59a_42bc_9cf9_f4662ea24771.slice/crio-0571ea52b55bcb568d9ef31a39b55adfd5337b08a733d6a3b9d71fb88f8faf43 WatchSource:0}: Error finding container 0571ea52b55bcb568d9ef31a39b55adfd5337b08a733d6a3b9d71fb88f8faf43: Status 404 returned error can't find the container with id 0571ea52b55bcb568d9ef31a39b55adfd5337b08a733d6a3b9d71fb88f8faf43
Oct 30 00:41:16.856343 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:16.856312    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p" event={"ID":"8ac3f48f-f59a-42bc-9cf9-f4662ea24771","Type":"ContainerStarted","Data":"0571ea52b55bcb568d9ef31a39b55adfd5337b08a733d6a3b9d71fb88f8faf43"}
Oct 30 00:41:18.275000 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:18.274906    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p"]
Oct 30 00:41:18.276338 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:18.275818    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p"]
Oct 30 00:41:18.327060 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:18.326910    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff"]
Oct 30 00:41:18.334628 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:18.334604    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff"
Oct 30 00:41:18.358974 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:18.358901    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff"]
Oct 30 00:41:18.415287 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:18.415255    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2bae925f-1ebd-4120-92e3-c93791012627-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-khkff\" (UID: \"2bae925f-1ebd-4120-92e3-c93791012627\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff"
Oct 30 00:41:18.415287 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:18.415291    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2clcq\" (UniqueName: \"kubernetes.io/projected/2bae925f-1ebd-4120-92e3-c93791012627-kube-api-access-2clcq\") pod \"kuadrant-operator-controller-manager-956956bc4-khkff\" (UID: \"2bae925f-1ebd-4120-92e3-c93791012627\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff"
Oct 30 00:41:18.516576 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:18.516536    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2bae925f-1ebd-4120-92e3-c93791012627-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-khkff\" (UID: \"2bae925f-1ebd-4120-92e3-c93791012627\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff"
Oct 30 00:41:18.516576 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:18.516584    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2clcq\" (UniqueName: \"kubernetes.io/projected/2bae925f-1ebd-4120-92e3-c93791012627-kube-api-access-2clcq\") pod \"kuadrant-operator-controller-manager-956956bc4-khkff\" (UID: \"2bae925f-1ebd-4120-92e3-c93791012627\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff"
Oct 30 00:41:18.516996 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:18.516960    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2bae925f-1ebd-4120-92e3-c93791012627-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-khkff\" (UID: \"2bae925f-1ebd-4120-92e3-c93791012627\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff"
Oct 30 00:41:18.541038 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:18.541010    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2clcq\" (UniqueName: \"kubernetes.io/projected/2bae925f-1ebd-4120-92e3-c93791012627-kube-api-access-2clcq\") pod \"kuadrant-operator-controller-manager-956956bc4-khkff\" (UID: \"2bae925f-1ebd-4120-92e3-c93791012627\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff"
Oct 30 00:41:18.653116 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:18.653080    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff"
Oct 30 00:41:18.989912 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:18.989839    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff"]
Oct 30 00:41:18.994086 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:41:18.994060    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bae925f_1ebd_4120_92e3_c93791012627.slice/crio-196f838eaf02583802f76c702c8db7c6246df2c5b7be0679a22f0f7f94997463 WatchSource:0}: Error finding container 196f838eaf02583802f76c702c8db7c6246df2c5b7be0679a22f0f7f94997463: Status 404 returned error can't find the container with id 196f838eaf02583802f76c702c8db7c6246df2c5b7be0679a22f0f7f94997463
Oct 30 00:41:19.880460 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:19.880417    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff" event={"ID":"2bae925f-1ebd-4120-92e3-c93791012627","Type":"ContainerStarted","Data":"196f838eaf02583802f76c702c8db7c6246df2c5b7be0679a22f0f7f94997463"}
Oct 30 00:41:21.891823 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:21.891789    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p" podUID="8ac3f48f-f59a-42bc-9cf9-f4662ea24771" containerName="manager" containerID="cri-o://8a1645cd107590d6ea602494ba7b0127db5d7f0f4e8462e3c90c81fa387a67a5" gracePeriod=2
Oct 30 00:41:21.893995 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:21.893965    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff" event={"ID":"2bae925f-1ebd-4120-92e3-c93791012627","Type":"ContainerStarted","Data":"a3c34270fdc96e950f087647f0acb3521982ded81ffe0d9004a7e2e7e511275e"}
Oct 30 00:41:21.894112 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:21.894071    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff"
Oct 30 00:41:21.957552 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:21.957505    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff" podStartSLOduration=1.243776694 podStartE2EDuration="3.957492424s" totalImagesPullingTime="2.71371573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:21.946911163 +0000 UTC m=+375.803818917" watchObservedRunningTime="2026-10-30 00:41:21.957492424 +0000 UTC m=+375.814400178"
Oct 30 00:41:22.107927 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:22.107904    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p"
Oct 30 00:41:22.119276 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:41:22.119253    6099 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-t6v5p\" is forbidden: User \"system:node:ip-10-0-130-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-245.ec2.internal' and this object" podUID="8ac3f48f-f59a-42bc-9cf9-f4662ea24771" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p"
Oct 30 00:41:22.149742 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:22.149722    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac3f48f-f59a-42bc-9cf9-f4662ea24771-kube-api-access-6t24x\" (UniqueName: \"kubernetes.io/projected/8ac3f48f-f59a-42bc-9cf9-f4662ea24771-kube-api-access-6t24x\") pod \"8ac3f48f-f59a-42bc-9cf9-f4662ea24771\" (UID: \"8ac3f48f-f59a-42bc-9cf9-f4662ea24771\") "
Oct 30 00:41:22.149814 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:22.149771    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac3f48f-f59a-42bc-9cf9-f4662ea24771-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ac3f48f-f59a-42bc-9cf9-f4662ea24771-extensions-socket-volume\") pod \"8ac3f48f-f59a-42bc-9cf9-f4662ea24771\" (UID: \"8ac3f48f-f59a-42bc-9cf9-f4662ea24771\") "
Oct 30 00:41:22.150041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:22.150020    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac3f48f-f59a-42bc-9cf9-f4662ea24771-extensions-socket-volume" pod "8ac3f48f-f59a-42bc-9cf9-f4662ea24771" (UID: "8ac3f48f-f59a-42bc-9cf9-f4662ea24771"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:41:22.151742 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:22.151701    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac3f48f-f59a-42bc-9cf9-f4662ea24771-kube-api-access-6t24x" pod "8ac3f48f-f59a-42bc-9cf9-f4662ea24771" (UID: "8ac3f48f-f59a-42bc-9cf9-f4662ea24771"). InnerVolumeSpecName "kube-api-access-6t24x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:41:22.250582 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:22.250519    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t24x\" (UniqueName: \"kubernetes.io/projected/8ac3f48f-f59a-42bc-9cf9-f4662ea24771-kube-api-access-6t24x\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:41:22.250582 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:22.250546    6099 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ac3f48f-f59a-42bc-9cf9-f4662ea24771-extensions-socket-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:41:22.259927 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:22.259900    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac3f48f-f59a-42bc-9cf9-f4662ea24771" path="/var/lib/kubelet/pods/8ac3f48f-f59a-42bc-9cf9-f4662ea24771/volumes"
Oct 30 00:41:22.899356 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:22.899317    6099 generic.go:356] "Generic (PLEG): container finished" podID="8ac3f48f-f59a-42bc-9cf9-f4662ea24771" containerID="8a1645cd107590d6ea602494ba7b0127db5d7f0f4e8462e3c90c81fa387a67a5" exitCode=2
Oct 30 00:41:22.899777 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:22.899368    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p"
Oct 30 00:41:22.899777 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:22.899414    6099 scope.go:122] "RemoveContainer" containerID="8a1645cd107590d6ea602494ba7b0127db5d7f0f4e8462e3c90c81fa387a67a5"
Oct 30 00:41:22.907660 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:22.907643    6099 scope.go:122] "RemoveContainer" containerID="8a1645cd107590d6ea602494ba7b0127db5d7f0f4e8462e3c90c81fa387a67a5"
Oct 30 00:41:22.907950 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:41:22.907932    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a1645cd107590d6ea602494ba7b0127db5d7f0f4e8462e3c90c81fa387a67a5\": container with ID starting with 8a1645cd107590d6ea602494ba7b0127db5d7f0f4e8462e3c90c81fa387a67a5 not found: ID does not exist" containerID="8a1645cd107590d6ea602494ba7b0127db5d7f0f4e8462e3c90c81fa387a67a5"
Oct 30 00:41:22.907996 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:22.907961    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a1645cd107590d6ea602494ba7b0127db5d7f0f4e8462e3c90c81fa387a67a5"} err="failed to get container status \"8a1645cd107590d6ea602494ba7b0127db5d7f0f4e8462e3c90c81fa387a67a5\": rpc error: code = NotFound desc = could not find container \"8a1645cd107590d6ea602494ba7b0127db5d7f0f4e8462e3c90c81fa387a67a5\": container with ID starting with 8a1645cd107590d6ea602494ba7b0127db5d7f0f4e8462e3c90c81fa387a67a5 not found: ID does not exist"
Oct 30 00:41:22.913105 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:41:22.913084    6099 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-t6v5p\" is forbidden: User \"system:node:ip-10-0-130-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-245.ec2.internal' and this object" podUID="8ac3f48f-f59a-42bc-9cf9-f4662ea24771" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-t6v5p"
Oct 30 00:41:32.902072 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:32.902038    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff"
Oct 30 00:41:45.984920 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:45.984886    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff"]
Oct 30 00:41:45.985527 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:45.985165    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff" podUID="2bae925f-1ebd-4120-92e3-c93791012627" containerName="manager" containerID="cri-o://a3c34270fdc96e950f087647f0acb3521982ded81ffe0d9004a7e2e7e511275e" gracePeriod=10
Oct 30 00:41:46.627759 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:41:46.627725    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:41:46.761283 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:46.761258    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff"
Oct 30 00:41:46.878510 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:46.878440    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bae925f-1ebd-4120-92e3-c93791012627-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2bae925f-1ebd-4120-92e3-c93791012627-extensions-socket-volume\") pod \"2bae925f-1ebd-4120-92e3-c93791012627\" (UID: \"2bae925f-1ebd-4120-92e3-c93791012627\") "
Oct 30 00:41:46.878510 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:46.878481    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bae925f-1ebd-4120-92e3-c93791012627-kube-api-access-2clcq\" (UniqueName: \"kubernetes.io/projected/2bae925f-1ebd-4120-92e3-c93791012627-kube-api-access-2clcq\") pod \"2bae925f-1ebd-4120-92e3-c93791012627\" (UID: \"2bae925f-1ebd-4120-92e3-c93791012627\") "
Oct 30 00:41:46.879007 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:46.878981    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bae925f-1ebd-4120-92e3-c93791012627-extensions-socket-volume" pod "2bae925f-1ebd-4120-92e3-c93791012627" (UID: "2bae925f-1ebd-4120-92e3-c93791012627"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:41:46.880581 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:46.880559    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bae925f-1ebd-4120-92e3-c93791012627-kube-api-access-2clcq" pod "2bae925f-1ebd-4120-92e3-c93791012627" (UID: "2bae925f-1ebd-4120-92e3-c93791012627"). InnerVolumeSpecName "kube-api-access-2clcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:41:46.979088 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:46.979057    6099 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2bae925f-1ebd-4120-92e3-c93791012627-extensions-socket-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:41:46.979088 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:46.979082    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2clcq\" (UniqueName: \"kubernetes.io/projected/2bae925f-1ebd-4120-92e3-c93791012627-kube-api-access-2clcq\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:41:47.005013 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:47.004976    6099 generic.go:356] "Generic (PLEG): container finished" podID="2bae925f-1ebd-4120-92e3-c93791012627" containerID="a3c34270fdc96e950f087647f0acb3521982ded81ffe0d9004a7e2e7e511275e" exitCode=0
Oct 30 00:41:47.005416 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:47.005033    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff"
Oct 30 00:41:47.005416 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:47.005069    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff" event={"ID":"2bae925f-1ebd-4120-92e3-c93791012627","Type":"ContainerDied","Data":"a3c34270fdc96e950f087647f0acb3521982ded81ffe0d9004a7e2e7e511275e"}
Oct 30 00:41:47.005416 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:47.005125    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff" event={"ID":"2bae925f-1ebd-4120-92e3-c93791012627","Type":"ContainerDied","Data":"196f838eaf02583802f76c702c8db7c6246df2c5b7be0679a22f0f7f94997463"}
Oct 30 00:41:47.005416 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:47.005151    6099 scope.go:122] "RemoveContainer" containerID="a3c34270fdc96e950f087647f0acb3521982ded81ffe0d9004a7e2e7e511275e"
Oct 30 00:41:47.013931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:47.013915    6099 scope.go:122] "RemoveContainer" containerID="a3c34270fdc96e950f087647f0acb3521982ded81ffe0d9004a7e2e7e511275e"
Oct 30 00:41:47.014150 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:41:47.014135    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3c34270fdc96e950f087647f0acb3521982ded81ffe0d9004a7e2e7e511275e\": container with ID starting with a3c34270fdc96e950f087647f0acb3521982ded81ffe0d9004a7e2e7e511275e not found: ID does not exist" containerID="a3c34270fdc96e950f087647f0acb3521982ded81ffe0d9004a7e2e7e511275e"
Oct 30 00:41:47.014197 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:47.014157    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3c34270fdc96e950f087647f0acb3521982ded81ffe0d9004a7e2e7e511275e"} err="failed to get container status \"a3c34270fdc96e950f087647f0acb3521982ded81ffe0d9004a7e2e7e511275e\": rpc error: code = NotFound desc = could not find container \"a3c34270fdc96e950f087647f0acb3521982ded81ffe0d9004a7e2e7e511275e\": container with ID starting with a3c34270fdc96e950f087647f0acb3521982ded81ffe0d9004a7e2e7e511275e not found: ID does not exist"
Oct 30 00:41:47.053999 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:47.053973    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff"]
Oct 30 00:41:47.060035 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:47.060016    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-khkff"]
Oct 30 00:41:48.259383 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:41:48.259345    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bae925f-1ebd-4120-92e3-c93791012627" path="/var/lib/kubelet/pods/2bae925f-1ebd-4120-92e3-c93791012627/volumes"
Oct 30 00:42:06.100610 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.100583    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nswtm"]
Oct 30 00:42:06.101099 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.100999    6099 cpu_manager.go:414] "containerMap: removing container" podUID="2bae925f-1ebd-4120-92e3-c93791012627" containerName="manager"
Oct 30 00:42:06.101099 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.101011    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bae925f-1ebd-4120-92e3-c93791012627" containerName="manager"
Oct 30 00:42:06.101099 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.101025    6099 cpu_manager.go:414] "containerMap: removing container" podUID="8ac3f48f-f59a-42bc-9cf9-f4662ea24771" containerName="manager"
Oct 30 00:42:06.101099 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.101031    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac3f48f-f59a-42bc-9cf9-f4662ea24771" containerName="manager"
Oct 30 00:42:06.101231 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.101111    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bae925f-1ebd-4120-92e3-c93791012627" containerName="manager"
Oct 30 00:42:06.101231 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.101120    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac3f48f-f59a-42bc-9cf9-f4662ea24771" containerName="manager"
Oct 30 00:42:06.104526 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.104507    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nswtm"
Oct 30 00:42:06.120480 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.120457    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nswtm"]
Oct 30 00:42:06.125364 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.125339    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8h6tt\""
Oct 30 00:42:06.140789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.140767    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcwsr\" (UniqueName: \"kubernetes.io/projected/79343130-4932-408f-aef2-33d8b86066c3-kube-api-access-pcwsr\") pod \"authorino-57f7f8659f-nswtm\" (UID: \"79343130-4932-408f-aef2-33d8b86066c3\") " pod="kuadrant-system/authorino-57f7f8659f-nswtm"
Oct 30 00:42:06.241996 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.241968    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcwsr\" (UniqueName: \"kubernetes.io/projected/79343130-4932-408f-aef2-33d8b86066c3-kube-api-access-pcwsr\") pod \"authorino-57f7f8659f-nswtm\" (UID: \"79343130-4932-408f-aef2-33d8b86066c3\") " pod="kuadrant-system/authorino-57f7f8659f-nswtm"
Oct 30 00:42:06.265046 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.265019    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcwsr\" (UniqueName: \"kubernetes.io/projected/79343130-4932-408f-aef2-33d8b86066c3-kube-api-access-pcwsr\") pod \"authorino-57f7f8659f-nswtm\" (UID: \"79343130-4932-408f-aef2-33d8b86066c3\") " pod="kuadrant-system/authorino-57f7f8659f-nswtm"
Oct 30 00:42:06.427548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.427482    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8h6tt\""
Oct 30 00:42:06.436521 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.436502    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nswtm"
Oct 30 00:42:06.696286 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.696214    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2vltw"]
Oct 30 00:42:06.700828 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.700805    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2vltw"
Oct 30 00:42:06.702603 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.702581    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:42:06.705913 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.705896    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"
Oct 30 00:42:06.718776 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.718757    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2vltw"]
Oct 30 00:42:06.721162 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.721147    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 00:42:06.725038 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.725014    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:42:06.731017 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.730913    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nswtm"]
Oct 30 00:42:06.733782 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:42:06.733755    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79343130_4932_408f_aef2_33d8b86066c3.slice/crio-30faada964f4becf5c8feaff7e1570d8cc35914c382e80fad493a1c2b91ac624 WatchSource:0}: Error finding container 30faada964f4becf5c8feaff7e1570d8cc35914c382e80fad493a1c2b91ac624: Status 404 returned error can't find the container with id 30faada964f4becf5c8feaff7e1570d8cc35914c382e80fad493a1c2b91ac624
Oct 30 00:42:06.746492 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.746468    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4zr9\" (UniqueName: \"kubernetes.io/projected/ea85e75b-50f3-4343-b254-749f59cf506c-kube-api-access-n4zr9\") pod \"authorino-56c6c9ddd5-2vltw\" (UID: \"ea85e75b-50f3-4343-b254-749f59cf506c\") " pod="kuadrant-system/authorino-56c6c9ddd5-2vltw"
Oct 30 00:42:06.746565 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.746507    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d7baecc5-e3d8-46e8-9ec1-883d47e82c85-config-file\") pod \"limitador-limitador-7c9b87d574-2mqnz\" (UID: \"d7baecc5-e3d8-46e8-9ec1-883d47e82c85\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"
Oct 30 00:42:06.746565 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.746527    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svbbs\" (UniqueName: \"kubernetes.io/projected/d7baecc5-e3d8-46e8-9ec1-883d47e82c85-kube-api-access-svbbs\") pod \"limitador-limitador-7c9b87d574-2mqnz\" (UID: \"d7baecc5-e3d8-46e8-9ec1-883d47e82c85\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"
Oct 30 00:42:06.791829 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.791803    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:42:06.847060 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.847028    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4zr9\" (UniqueName: \"kubernetes.io/projected/ea85e75b-50f3-4343-b254-749f59cf506c-kube-api-access-n4zr9\") pod \"authorino-56c6c9ddd5-2vltw\" (UID: \"ea85e75b-50f3-4343-b254-749f59cf506c\") " pod="kuadrant-system/authorino-56c6c9ddd5-2vltw"
Oct 30 00:42:06.847201 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.847076    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d7baecc5-e3d8-46e8-9ec1-883d47e82c85-config-file\") pod \"limitador-limitador-7c9b87d574-2mqnz\" (UID: \"d7baecc5-e3d8-46e8-9ec1-883d47e82c85\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"
Oct 30 00:42:06.847264 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.847206    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svbbs\" (UniqueName: \"kubernetes.io/projected/d7baecc5-e3d8-46e8-9ec1-883d47e82c85-kube-api-access-svbbs\") pod \"limitador-limitador-7c9b87d574-2mqnz\" (UID: \"d7baecc5-e3d8-46e8-9ec1-883d47e82c85\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"
Oct 30 00:42:06.847698 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.847676    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d7baecc5-e3d8-46e8-9ec1-883d47e82c85-config-file\") pod \"limitador-limitador-7c9b87d574-2mqnz\" (UID: \"d7baecc5-e3d8-46e8-9ec1-883d47e82c85\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"
Oct 30 00:42:06.868340 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.868308    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svbbs\" (UniqueName: \"kubernetes.io/projected/d7baecc5-e3d8-46e8-9ec1-883d47e82c85-kube-api-access-svbbs\") pod \"limitador-limitador-7c9b87d574-2mqnz\" (UID: \"d7baecc5-e3d8-46e8-9ec1-883d47e82c85\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"
Oct 30 00:42:06.868423 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:06.868354    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4zr9\" (UniqueName: \"kubernetes.io/projected/ea85e75b-50f3-4343-b254-749f59cf506c-kube-api-access-n4zr9\") pod \"authorino-56c6c9ddd5-2vltw\" (UID: \"ea85e75b-50f3-4343-b254-749f59cf506c\") " pod="kuadrant-system/authorino-56c6c9ddd5-2vltw"
Oct 30 00:42:07.014766 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:07.014697    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2vltw"
Oct 30 00:42:07.020604 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:07.020582    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"
Oct 30 00:42:07.092037 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:07.092005    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nswtm" event={"ID":"79343130-4932-408f-aef2-33d8b86066c3","Type":"ContainerStarted","Data":"30faada964f4becf5c8feaff7e1570d8cc35914c382e80fad493a1c2b91ac624"}
Oct 30 00:42:07.308276 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:07.308243    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2vltw"]
Oct 30 00:42:07.311325 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:42:07.311298    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea85e75b_50f3_4343_b254_749f59cf506c.slice/crio-f91a5634110bf8616ad7242d931a7e92cf1683f2f7216a5d472fad9c0b9d37ca WatchSource:0}: Error finding container f91a5634110bf8616ad7242d931a7e92cf1683f2f7216a5d472fad9c0b9d37ca: Status 404 returned error can't find the container with id f91a5634110bf8616ad7242d931a7e92cf1683f2f7216a5d472fad9c0b9d37ca
Oct 30 00:42:07.321329 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:07.321302    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:42:07.324672 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:42:07.324634    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7baecc5_e3d8_46e8_9ec1_883d47e82c85.slice/crio-f729114029472e39634c7f2086fd0f6a879cf909546aa370a1367bc5418e0fb3 WatchSource:0}: Error finding container f729114029472e39634c7f2086fd0f6a879cf909546aa370a1367bc5418e0fb3: Status 404 returned error can't find the container with id f729114029472e39634c7f2086fd0f6a879cf909546aa370a1367bc5418e0fb3
Oct 30 00:42:08.099254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:08.099211    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2vltw" event={"ID":"ea85e75b-50f3-4343-b254-749f59cf506c","Type":"ContainerStarted","Data":"f91a5634110bf8616ad7242d931a7e92cf1683f2f7216a5d472fad9c0b9d37ca"}
Oct 30 00:42:08.100575 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:08.100541    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-2mqnz" event={"ID":"d7baecc5-e3d8-46e8-9ec1-883d47e82c85","Type":"ContainerStarted","Data":"f729114029472e39634c7f2086fd0f6a879cf909546aa370a1367bc5418e0fb3"}
Oct 30 00:42:09.106878 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:09.106840    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2vltw" event={"ID":"ea85e75b-50f3-4343-b254-749f59cf506c","Type":"ContainerStarted","Data":"18d6b25950f82a240979d650bb107cc2b91445002afbba9eab8fe4310f31ee13"}
Oct 30 00:42:09.108606 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:09.108580    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-2mqnz" event={"ID":"d7baecc5-e3d8-46e8-9ec1-883d47e82c85","Type":"ContainerStarted","Data":"c86811c684e01c61af7541897204d8b12fa1abbdf69d144953b9451ff01e677b"}
Oct 30 00:42:09.108738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:09.108694    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"
Oct 30 00:42:09.110206 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:09.110180    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nswtm" event={"ID":"79343130-4932-408f-aef2-33d8b86066c3","Type":"ContainerStarted","Data":"6ced3c8961c0a9c2e6093408ba544b7762ac91a14ebaf2d38fafb08514533be0"}
Oct 30 00:42:09.140445 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:09.140397    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-2vltw" podStartSLOduration=1.419762778 podStartE2EDuration="3.140385548s" totalImagesPullingTime="1.72062277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:09.139903283 +0000 UTC m=+422.996811037" watchObservedRunningTime="2026-10-30 00:42:09.140385548 +0000 UTC m=+422.997293303"
Oct 30 00:42:09.164549 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:09.164516    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nswtm"]
Oct 30 00:42:09.173027 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:09.172980    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-nswtm" podStartSLOduration=1.292188054 podStartE2EDuration="3.172968659s" totalImagesPullingTime="1.880780605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:09.172830976 +0000 UTC m=+423.029738732" watchObservedRunningTime="2026-10-30 00:42:09.172968659 +0000 UTC m=+423.029876414"
Oct 30 00:42:09.208253 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:09.208198    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-2mqnz" podStartSLOduration=2.2529768519999998 podStartE2EDuration="3.208185952s" totalImagesPullingTime="955.2091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:09.208149284 +0000 UTC m=+423.065057049" watchObservedRunningTime="2026-10-30 00:42:09.208185952 +0000 UTC m=+423.065093704"
Oct 30 00:42:11.119259 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:11.119219    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-nswtm" podUID="79343130-4932-408f-aef2-33d8b86066c3" containerName="authorino" containerID="cri-o://6ced3c8961c0a9c2e6093408ba544b7762ac91a14ebaf2d38fafb08514533be0" gracePeriod=30
Oct 30 00:42:11.358182 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:11.358159    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nswtm"
Oct 30 00:42:11.488794 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:11.488696    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79343130-4932-408f-aef2-33d8b86066c3-kube-api-access-pcwsr\" (UniqueName: \"kubernetes.io/projected/79343130-4932-408f-aef2-33d8b86066c3-kube-api-access-pcwsr\") pod \"79343130-4932-408f-aef2-33d8b86066c3\" (UID: \"79343130-4932-408f-aef2-33d8b86066c3\") "
Oct 30 00:42:11.490732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:11.490679    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79343130-4932-408f-aef2-33d8b86066c3-kube-api-access-pcwsr" pod "79343130-4932-408f-aef2-33d8b86066c3" (UID: "79343130-4932-408f-aef2-33d8b86066c3"). InnerVolumeSpecName "kube-api-access-pcwsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:42:11.589845 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:11.589817    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcwsr\" (UniqueName: \"kubernetes.io/projected/79343130-4932-408f-aef2-33d8b86066c3-kube-api-access-pcwsr\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:42:12.124623 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:12.124589    6099 generic.go:356] "Generic (PLEG): container finished" podID="79343130-4932-408f-aef2-33d8b86066c3" containerID="6ced3c8961c0a9c2e6093408ba544b7762ac91a14ebaf2d38fafb08514533be0" exitCode=0
Oct 30 00:42:12.125046 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:12.124635    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nswtm"
Oct 30 00:42:12.125046 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:12.124676    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nswtm" event={"ID":"79343130-4932-408f-aef2-33d8b86066c3","Type":"ContainerDied","Data":"6ced3c8961c0a9c2e6093408ba544b7762ac91a14ebaf2d38fafb08514533be0"}
Oct 30 00:42:12.125046 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:12.124737    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nswtm" event={"ID":"79343130-4932-408f-aef2-33d8b86066c3","Type":"ContainerDied","Data":"30faada964f4becf5c8feaff7e1570d8cc35914c382e80fad493a1c2b91ac624"}
Oct 30 00:42:12.125046 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:12.124756    6099 scope.go:122] "RemoveContainer" containerID="6ced3c8961c0a9c2e6093408ba544b7762ac91a14ebaf2d38fafb08514533be0"
Oct 30 00:42:12.133152 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:12.133136    6099 scope.go:122] "RemoveContainer" containerID="6ced3c8961c0a9c2e6093408ba544b7762ac91a14ebaf2d38fafb08514533be0"
Oct 30 00:42:12.133429 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:42:12.133410    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ced3c8961c0a9c2e6093408ba544b7762ac91a14ebaf2d38fafb08514533be0\": container with ID starting with 6ced3c8961c0a9c2e6093408ba544b7762ac91a14ebaf2d38fafb08514533be0 not found: ID does not exist" containerID="6ced3c8961c0a9c2e6093408ba544b7762ac91a14ebaf2d38fafb08514533be0"
Oct 30 00:42:12.133491 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:12.133435    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ced3c8961c0a9c2e6093408ba544b7762ac91a14ebaf2d38fafb08514533be0"} err="failed to get container status \"6ced3c8961c0a9c2e6093408ba544b7762ac91a14ebaf2d38fafb08514533be0\": rpc error: code = NotFound desc = could not find container \"6ced3c8961c0a9c2e6093408ba544b7762ac91a14ebaf2d38fafb08514533be0\": container with ID starting with 6ced3c8961c0a9c2e6093408ba544b7762ac91a14ebaf2d38fafb08514533be0 not found: ID does not exist"
Oct 30 00:42:12.166998 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:12.166972    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nswtm"]
Oct 30 00:42:12.175421 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:12.175403    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nswtm"]
Oct 30 00:42:12.259480 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:12.259449    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79343130-4932-408f-aef2-33d8b86066c3" path="/var/lib/kubelet/pods/79343130-4932-408f-aef2-33d8b86066c3/volumes"
Oct 30 00:42:16.627107 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:42:16.627068    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:42:20.116331 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:20.116303    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"
Oct 30 00:42:40.153448 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:40.153416    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56d59d5b8c-pf2rm"]
Oct 30 00:42:40.153937 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:40.153843    6099 cpu_manager.go:414] "containerMap: removing container" podUID="79343130-4932-408f-aef2-33d8b86066c3" containerName="authorino"
Oct 30 00:42:40.153937 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:40.153855    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79343130-4932-408f-aef2-33d8b86066c3" containerName="authorino"
Oct 30 00:42:40.153937 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:40.153925    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="79343130-4932-408f-aef2-33d8b86066c3" containerName="authorino"
Oct 30 00:42:40.158348 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:40.158325    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56d59d5b8c-pf2rm"
Oct 30 00:42:40.175603 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:40.175573    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56d59d5b8c-pf2rm"]
Oct 30 00:42:40.184842 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:40.184822    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 00:42:40.249638 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:40.249605    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e699c8a-dfe8-47d2-ba1f-8ebce73d7121-tls-cert\") pod \"authorino-56d59d5b8c-pf2rm\" (UID: \"9e699c8a-dfe8-47d2-ba1f-8ebce73d7121\") " pod="kuadrant-system/authorino-56d59d5b8c-pf2rm"
Oct 30 00:42:40.249638 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:40.249640    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk2bb\" (UniqueName: \"kubernetes.io/projected/9e699c8a-dfe8-47d2-ba1f-8ebce73d7121-kube-api-access-pk2bb\") pod \"authorino-56d59d5b8c-pf2rm\" (UID: \"9e699c8a-dfe8-47d2-ba1f-8ebce73d7121\") " pod="kuadrant-system/authorino-56d59d5b8c-pf2rm"
Oct 30 00:42:40.350773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:40.350737    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e699c8a-dfe8-47d2-ba1f-8ebce73d7121-tls-cert\") pod \"authorino-56d59d5b8c-pf2rm\" (UID: \"9e699c8a-dfe8-47d2-ba1f-8ebce73d7121\") " pod="kuadrant-system/authorino-56d59d5b8c-pf2rm"
Oct 30 00:42:40.350926 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:40.350780    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk2bb\" (UniqueName: \"kubernetes.io/projected/9e699c8a-dfe8-47d2-ba1f-8ebce73d7121-kube-api-access-pk2bb\") pod \"authorino-56d59d5b8c-pf2rm\" (UID: \"9e699c8a-dfe8-47d2-ba1f-8ebce73d7121\") " pod="kuadrant-system/authorino-56d59d5b8c-pf2rm"
Oct 30 00:42:40.353181 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:40.353159    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e699c8a-dfe8-47d2-ba1f-8ebce73d7121-tls-cert\") pod \"authorino-56d59d5b8c-pf2rm\" (UID: \"9e699c8a-dfe8-47d2-ba1f-8ebce73d7121\") " pod="kuadrant-system/authorino-56d59d5b8c-pf2rm"
Oct 30 00:42:40.371425 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:40.371396    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk2bb\" (UniqueName: \"kubernetes.io/projected/9e699c8a-dfe8-47d2-ba1f-8ebce73d7121-kube-api-access-pk2bb\") pod \"authorino-56d59d5b8c-pf2rm\" (UID: \"9e699c8a-dfe8-47d2-ba1f-8ebce73d7121\") " pod="kuadrant-system/authorino-56d59d5b8c-pf2rm"
Oct 30 00:42:40.469455 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:40.469380    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56d59d5b8c-pf2rm"
Oct 30 00:42:40.760027 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:40.759962    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56d59d5b8c-pf2rm"]
Oct 30 00:42:40.763605 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:42:40.763578    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e699c8a_dfe8_47d2_ba1f_8ebce73d7121.slice/crio-fa9e4773b33072ddb39dc9f74f93799f96b83b089a25b8f06dac86f50c8df3af WatchSource:0}: Error finding container fa9e4773b33072ddb39dc9f74f93799f96b83b089a25b8f06dac86f50c8df3af: Status 404 returned error can't find the container with id fa9e4773b33072ddb39dc9f74f93799f96b83b089a25b8f06dac86f50c8df3af
Oct 30 00:42:41.254019 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:41.253989    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56d59d5b8c-pf2rm" event={"ID":"9e699c8a-dfe8-47d2-ba1f-8ebce73d7121","Type":"ContainerStarted","Data":"92033c06cd82fb3d97ce81ea40be9f0e65fad9773e495be3ccfa77f353df1846"}
Oct 30 00:42:41.254385 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:41.254027    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56d59d5b8c-pf2rm" event={"ID":"9e699c8a-dfe8-47d2-ba1f-8ebce73d7121","Type":"ContainerStarted","Data":"fa9e4773b33072ddb39dc9f74f93799f96b83b089a25b8f06dac86f50c8df3af"}
Oct 30 00:42:41.288776 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:41.288733    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56d59d5b8c-pf2rm" podStartSLOduration=0.925896023 podStartE2EDuration="1.288702447s" totalImagesPullingTime="362.806424ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:41.287734159 +0000 UTC m=+455.144641907" watchObservedRunningTime="2026-10-30 00:42:41.288702447 +0000 UTC m=+455.145610242"
Oct 30 00:42:41.314675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:41.314628    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2vltw"]
Oct 30 00:42:41.314850 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:41.314831    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-2vltw" podUID="ea85e75b-50f3-4343-b254-749f59cf506c" containerName="authorino" containerID="cri-o://18d6b25950f82a240979d650bb107cc2b91445002afbba9eab8fe4310f31ee13" gracePeriod=30
Oct 30 00:42:41.555341 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:41.555317    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2vltw"
Oct 30 00:42:41.664547 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:41.664517    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea85e75b-50f3-4343-b254-749f59cf506c-kube-api-access-n4zr9\" (UniqueName: \"kubernetes.io/projected/ea85e75b-50f3-4343-b254-749f59cf506c-kube-api-access-n4zr9\") pod \"ea85e75b-50f3-4343-b254-749f59cf506c\" (UID: \"ea85e75b-50f3-4343-b254-749f59cf506c\") "
Oct 30 00:42:41.666537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:41.666515    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea85e75b-50f3-4343-b254-749f59cf506c-kube-api-access-n4zr9" pod "ea85e75b-50f3-4343-b254-749f59cf506c" (UID: "ea85e75b-50f3-4343-b254-749f59cf506c"). InnerVolumeSpecName "kube-api-access-n4zr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:42:41.765641 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:41.765578    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4zr9\" (UniqueName: \"kubernetes.io/projected/ea85e75b-50f3-4343-b254-749f59cf506c-kube-api-access-n4zr9\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:42:42.266261 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:42.266223    6099 generic.go:356] "Generic (PLEG): container finished" podID="ea85e75b-50f3-4343-b254-749f59cf506c" containerID="18d6b25950f82a240979d650bb107cc2b91445002afbba9eab8fe4310f31ee13" exitCode=0
Oct 30 00:42:42.266261 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:42.266270    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2vltw"
Oct 30 00:42:42.266814 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:42.266301    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2vltw" event={"ID":"ea85e75b-50f3-4343-b254-749f59cf506c","Type":"ContainerDied","Data":"18d6b25950f82a240979d650bb107cc2b91445002afbba9eab8fe4310f31ee13"}
Oct 30 00:42:42.266814 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:42.266341    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2vltw" event={"ID":"ea85e75b-50f3-4343-b254-749f59cf506c","Type":"ContainerDied","Data":"f91a5634110bf8616ad7242d931a7e92cf1683f2f7216a5d472fad9c0b9d37ca"}
Oct 30 00:42:42.266814 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:42.266362    6099 scope.go:122] "RemoveContainer" containerID="18d6b25950f82a240979d650bb107cc2b91445002afbba9eab8fe4310f31ee13"
Oct 30 00:42:42.275885 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:42.275868    6099 scope.go:122] "RemoveContainer" containerID="18d6b25950f82a240979d650bb107cc2b91445002afbba9eab8fe4310f31ee13"
Oct 30 00:42:42.276135 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:42:42.276116    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d6b25950f82a240979d650bb107cc2b91445002afbba9eab8fe4310f31ee13\": container with ID starting with 18d6b25950f82a240979d650bb107cc2b91445002afbba9eab8fe4310f31ee13 not found: ID does not exist" containerID="18d6b25950f82a240979d650bb107cc2b91445002afbba9eab8fe4310f31ee13"
Oct 30 00:42:42.276180 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:42.276142    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d6b25950f82a240979d650bb107cc2b91445002afbba9eab8fe4310f31ee13"} err="failed to get container status \"18d6b25950f82a240979d650bb107cc2b91445002afbba9eab8fe4310f31ee13\": rpc error: code = NotFound desc = could not find container \"18d6b25950f82a240979d650bb107cc2b91445002afbba9eab8fe4310f31ee13\": container with ID starting with 18d6b25950f82a240979d650bb107cc2b91445002afbba9eab8fe4310f31ee13 not found: ID does not exist"
Oct 30 00:42:42.311867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:42.311838    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2vltw"]
Oct 30 00:42:42.321530 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:42.321508    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2vltw"]
Oct 30 00:42:44.259510 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:42:44.259480    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea85e75b-50f3-4343-b254-749f59cf506c" path="/var/lib/kubelet/pods/ea85e75b-50f3-4343-b254-749f59cf506c/volumes"
Oct 30 00:42:46.628688 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:42:46.628652    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:43:16.627594 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:43:16.627554    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:43:28.405556 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:43:28.405501    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:43:46.628556 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:43:46.628521    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:44:16.628300 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:44:16.628188    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:44:31.192729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:31.190675    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:44:32.367419 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.367389    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"]
Oct 30 00:44:32.367906 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.367827    6099 cpu_manager.go:414] "containerMap: removing container" podUID="ea85e75b-50f3-4343-b254-749f59cf506c" containerName="authorino"
Oct 30 00:44:32.367906 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.367841    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea85e75b-50f3-4343-b254-749f59cf506c" containerName="authorino"
Oct 30 00:44:32.368019 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.367920    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea85e75b-50f3-4343-b254-749f59cf506c" containerName="authorino"
Oct 30 00:44:32.371102 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.371075    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.394473 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.394451    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 00:44:32.394473 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.394467    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 00:44:32.394734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.394547    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 00:44:32.394734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.394594    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-28sms\""
Oct 30 00:44:32.398847 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.398824    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"]
Oct 30 00:44:32.403263 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.403233    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x5lc\" (UniqueName: \"kubernetes.io/projected/4f0e2175-e7cb-48f7-b2ac-1c18e1890cde-kube-api-access-8x5lc\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hknzb\" (UID: \"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.403385 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.403365    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f0e2175-e7cb-48f7-b2ac-1c18e1890cde-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hknzb\" (UID: \"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.403453 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.403433    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f0e2175-e7cb-48f7-b2ac-1c18e1890cde-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hknzb\" (UID: \"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.403489 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.403468    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f0e2175-e7cb-48f7-b2ac-1c18e1890cde-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hknzb\" (UID: \"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.403532 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.403493    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f0e2175-e7cb-48f7-b2ac-1c18e1890cde-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hknzb\" (UID: \"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.403532 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.403516    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0e2175-e7cb-48f7-b2ac-1c18e1890cde-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hknzb\" (UID: \"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.504053 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.504019    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f0e2175-e7cb-48f7-b2ac-1c18e1890cde-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hknzb\" (UID: \"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.504248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.504058    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f0e2175-e7cb-48f7-b2ac-1c18e1890cde-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hknzb\" (UID: \"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.504248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.504079    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f0e2175-e7cb-48f7-b2ac-1c18e1890cde-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hknzb\" (UID: \"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.504248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.504111    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f0e2175-e7cb-48f7-b2ac-1c18e1890cde-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hknzb\" (UID: \"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.504248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.504144    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0e2175-e7cb-48f7-b2ac-1c18e1890cde-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hknzb\" (UID: \"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.504248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.504177    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5lc\" (UniqueName: \"kubernetes.io/projected/4f0e2175-e7cb-48f7-b2ac-1c18e1890cde-kube-api-access-8x5lc\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hknzb\" (UID: \"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.504580 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.504546    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f0e2175-e7cb-48f7-b2ac-1c18e1890cde-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hknzb\" (UID: \"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.504738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.504614    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f0e2175-e7cb-48f7-b2ac-1c18e1890cde-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hknzb\" (UID: \"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.504738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.504633    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f0e2175-e7cb-48f7-b2ac-1c18e1890cde-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hknzb\" (UID: \"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.506490 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.506467    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f0e2175-e7cb-48f7-b2ac-1c18e1890cde-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hknzb\" (UID: \"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.506739 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.506722    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0e2175-e7cb-48f7-b2ac-1c18e1890cde-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hknzb\" (UID: \"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.525660 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.525630    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x5lc\" (UniqueName: \"kubernetes.io/projected/4f0e2175-e7cb-48f7-b2ac-1c18e1890cde-kube-api-access-8x5lc\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hknzb\" (UID: \"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.686446 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.686368    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:32.980560 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.980427    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"]
Oct 30 00:44:32.983308 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:44:32.983283    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0e2175_e7cb_48f7_b2ac_1c18e1890cde.slice/crio-52025bba9dcac0bf7d90063e3a4c22491fbcbc1c6862c60b6f3643d115708cb8 WatchSource:0}: Error finding container 52025bba9dcac0bf7d90063e3a4c22491fbcbc1c6862c60b6f3643d115708cb8: Status 404 returned error can't find the container with id 52025bba9dcac0bf7d90063e3a4c22491fbcbc1c6862c60b6f3643d115708cb8
Oct 30 00:44:32.984571 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:32.984555    6099 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:44:33.550978 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:33.550940    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:44:33.758438 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:33.758406    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb" event={"ID":"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde","Type":"ContainerStarted","Data":"52025bba9dcac0bf7d90063e3a4c22491fbcbc1c6862c60b6f3643d115708cb8"}
Oct 30 00:44:34.764243 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:34.764212    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb" event={"ID":"4f0e2175-e7cb-48f7-b2ac-1c18e1890cde","Type":"ContainerStarted","Data":"cb9d47d3961e62588d9ffd923d23152a26bdaa819d0c1bba01a9e30e24b34c19"}
Oct 30 00:44:34.804243 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:34.804189    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb" podStartSLOduration=1.609679947 podStartE2EDuration="2.804175192s" totalImagesPullingTime="1.194495245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:34.802904162 +0000 UTC m=+568.659811917" watchObservedRunningTime="2026-10-30 00:44:34.804175192 +0000 UTC m=+568.661082941"
Oct 30 00:44:35.687125 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:35.687095    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:37.247862 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.247825    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"]
Oct 30 00:44:37.252663 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.252440    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.290871 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.285006    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 00:44:37.290871 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.286524    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"]
Oct 30 00:44:37.360282 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.360247    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d94d18a3-84f0-4225-bf91-19ce9a058574-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm\" (UID: \"d94d18a3-84f0-4225-bf91-19ce9a058574\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.360282 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.360286    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d94d18a3-84f0-4225-bf91-19ce9a058574-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm\" (UID: \"d94d18a3-84f0-4225-bf91-19ce9a058574\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.360500 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.360314    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d94d18a3-84f0-4225-bf91-19ce9a058574-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm\" (UID: \"d94d18a3-84f0-4225-bf91-19ce9a058574\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.360500 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.360452    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94d18a3-84f0-4225-bf91-19ce9a058574-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm\" (UID: \"d94d18a3-84f0-4225-bf91-19ce9a058574\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.360577 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.360517    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flndx\" (UniqueName: \"kubernetes.io/projected/d94d18a3-84f0-4225-bf91-19ce9a058574-kube-api-access-flndx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm\" (UID: \"d94d18a3-84f0-4225-bf91-19ce9a058574\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.360611 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.360576    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d94d18a3-84f0-4225-bf91-19ce9a058574-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm\" (UID: \"d94d18a3-84f0-4225-bf91-19ce9a058574\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.461619 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.461571    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d94d18a3-84f0-4225-bf91-19ce9a058574-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm\" (UID: \"d94d18a3-84f0-4225-bf91-19ce9a058574\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.461619 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.461610    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d94d18a3-84f0-4225-bf91-19ce9a058574-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm\" (UID: \"d94d18a3-84f0-4225-bf91-19ce9a058574\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.461898 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.461634    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d94d18a3-84f0-4225-bf91-19ce9a058574-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm\" (UID: \"d94d18a3-84f0-4225-bf91-19ce9a058574\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.461898 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.461664    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94d18a3-84f0-4225-bf91-19ce9a058574-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm\" (UID: \"d94d18a3-84f0-4225-bf91-19ce9a058574\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.461898 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.461742    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flndx\" (UniqueName: \"kubernetes.io/projected/d94d18a3-84f0-4225-bf91-19ce9a058574-kube-api-access-flndx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm\" (UID: \"d94d18a3-84f0-4225-bf91-19ce9a058574\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.461898 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.461779    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d94d18a3-84f0-4225-bf91-19ce9a058574-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm\" (UID: \"d94d18a3-84f0-4225-bf91-19ce9a058574\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.462156 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.462128    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d94d18a3-84f0-4225-bf91-19ce9a058574-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm\" (UID: \"d94d18a3-84f0-4225-bf91-19ce9a058574\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.462224 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.462139    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94d18a3-84f0-4225-bf91-19ce9a058574-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm\" (UID: \"d94d18a3-84f0-4225-bf91-19ce9a058574\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.462224 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.462212    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d94d18a3-84f0-4225-bf91-19ce9a058574-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm\" (UID: \"d94d18a3-84f0-4225-bf91-19ce9a058574\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.463992 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.463960    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d94d18a3-84f0-4225-bf91-19ce9a058574-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm\" (UID: \"d94d18a3-84f0-4225-bf91-19ce9a058574\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.464133 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.464116    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d94d18a3-84f0-4225-bf91-19ce9a058574-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm\" (UID: \"d94d18a3-84f0-4225-bf91-19ce9a058574\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.483348 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.483318    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flndx\" (UniqueName: \"kubernetes.io/projected/d94d18a3-84f0-4225-bf91-19ce9a058574-kube-api-access-flndx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm\" (UID: \"d94d18a3-84f0-4225-bf91-19ce9a058574\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.622352 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.622322    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:37.914764 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:37.914586    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"]
Oct 30 00:44:37.917385 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:44:37.917358    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94d18a3_84f0_4225_bf91_19ce9a058574.slice/crio-ba8f5d9a662275619391ad844db71a0a1dce70633756eae1ccc41707a046cc3a WatchSource:0}: Error finding container ba8f5d9a662275619391ad844db71a0a1dce70633756eae1ccc41707a046cc3a: Status 404 returned error can't find the container with id ba8f5d9a662275619391ad844db71a0a1dce70633756eae1ccc41707a046cc3a
Oct 30 00:44:38.694909 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.694873    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"]
Oct 30 00:44:38.698929 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.698906    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:38.714288 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.714263    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 00:44:38.725946 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.725773    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"]
Oct 30 00:44:38.743121 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.743097    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:44:38.773949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.773911    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8179b0-96f4-4647-b926-5969b4afecd4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv\" (UID: \"5c8179b0-96f4-4647-b926-5969b4afecd4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:38.774115 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.773954    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8179b0-96f4-4647-b926-5969b4afecd4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv\" (UID: \"5c8179b0-96f4-4647-b926-5969b4afecd4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:38.774115 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.774029    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smmv5\" (UniqueName: \"kubernetes.io/projected/5c8179b0-96f4-4647-b926-5969b4afecd4-kube-api-access-smmv5\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv\" (UID: \"5c8179b0-96f4-4647-b926-5969b4afecd4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:38.774234 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.774118    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8179b0-96f4-4647-b926-5969b4afecd4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv\" (UID: \"5c8179b0-96f4-4647-b926-5969b4afecd4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:38.774234 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.774195    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8179b0-96f4-4647-b926-5969b4afecd4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv\" (UID: \"5c8179b0-96f4-4647-b926-5969b4afecd4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:38.774368 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.774244    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8179b0-96f4-4647-b926-5969b4afecd4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv\" (UID: \"5c8179b0-96f4-4647-b926-5969b4afecd4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:38.785591 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.785561    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm" event={"ID":"d94d18a3-84f0-4225-bf91-19ce9a058574","Type":"ContainerStarted","Data":"65223c9df0128cc21b6fb907adcac110c77b0c591c68c51ebe640967bd0cd817"}
Oct 30 00:44:38.785761 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.785609    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm" event={"ID":"d94d18a3-84f0-4225-bf91-19ce9a058574","Type":"ContainerStarted","Data":"ba8f5d9a662275619391ad844db71a0a1dce70633756eae1ccc41707a046cc3a"}
Oct 30 00:44:38.829672 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.829617    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm" podStartSLOduration=1.6297442850000001 podStartE2EDuration="1.829602315s" totalImagesPullingTime="199.85803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:38.82836151 +0000 UTC m=+572.685269264" watchObservedRunningTime="2026-10-30 00:44:38.829602315 +0000 UTC m=+572.686510071"
Oct 30 00:44:38.874890 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.874844    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8179b0-96f4-4647-b926-5969b4afecd4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv\" (UID: \"5c8179b0-96f4-4647-b926-5969b4afecd4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:38.874890 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.874892    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8179b0-96f4-4647-b926-5969b4afecd4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv\" (UID: \"5c8179b0-96f4-4647-b926-5969b4afecd4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:38.875138 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.874913    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smmv5\" (UniqueName: \"kubernetes.io/projected/5c8179b0-96f4-4647-b926-5969b4afecd4-kube-api-access-smmv5\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv\" (UID: \"5c8179b0-96f4-4647-b926-5969b4afecd4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:38.875138 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.874948    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8179b0-96f4-4647-b926-5969b4afecd4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv\" (UID: \"5c8179b0-96f4-4647-b926-5969b4afecd4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:38.875138 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.875040    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8179b0-96f4-4647-b926-5969b4afecd4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv\" (UID: \"5c8179b0-96f4-4647-b926-5969b4afecd4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:38.875138 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.875093    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8179b0-96f4-4647-b926-5969b4afecd4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv\" (UID: \"5c8179b0-96f4-4647-b926-5969b4afecd4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:38.875348 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.875320    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8179b0-96f4-4647-b926-5969b4afecd4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv\" (UID: \"5c8179b0-96f4-4647-b926-5969b4afecd4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:38.875403 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.875351    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8179b0-96f4-4647-b926-5969b4afecd4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv\" (UID: \"5c8179b0-96f4-4647-b926-5969b4afecd4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:38.875577 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.875551    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8179b0-96f4-4647-b926-5969b4afecd4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv\" (UID: \"5c8179b0-96f4-4647-b926-5969b4afecd4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:38.877693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.877664    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8179b0-96f4-4647-b926-5969b4afecd4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv\" (UID: \"5c8179b0-96f4-4647-b926-5969b4afecd4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:38.877902 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.877876    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8179b0-96f4-4647-b926-5969b4afecd4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv\" (UID: \"5c8179b0-96f4-4647-b926-5969b4afecd4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:38.896318 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:38.896293    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smmv5\" (UniqueName: \"kubernetes.io/projected/5c8179b0-96f4-4647-b926-5969b4afecd4-kube-api-access-smmv5\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv\" (UID: \"5c8179b0-96f4-4647-b926-5969b4afecd4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:39.015227 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:39.015148    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:39.316732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:39.316673    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"]
Oct 30 00:44:39.321001 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:44:39.320948    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8179b0_96f4_4647_b926_5969b4afecd4.slice/crio-5704b54feea5aeae32dac51479c369d91e4efc736dff5ac2fd352f549ccf491c WatchSource:0}: Error finding container 5704b54feea5aeae32dac51479c369d91e4efc736dff5ac2fd352f549ccf491c: Status 404 returned error can't find the container with id 5704b54feea5aeae32dac51479c369d91e4efc736dff5ac2fd352f549ccf491c
Oct 30 00:44:39.622467 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:39.622437    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:39.794672 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:39.794633    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv" event={"ID":"5c8179b0-96f4-4647-b926-5969b4afecd4","Type":"ContainerStarted","Data":"78c046f04f1f6125d3473d07fd653318d5296aa02761d65f3395452183721601"}
Oct 30 00:44:39.795083 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:39.794743    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv" event={"ID":"5c8179b0-96f4-4647-b926-5969b4afecd4","Type":"ContainerStarted","Data":"5704b54feea5aeae32dac51479c369d91e4efc736dff5ac2fd352f549ccf491c"}
Oct 30 00:44:39.833688 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:39.833596    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv" podStartSLOduration=1.615676256 podStartE2EDuration="1.833583434s" totalImagesPullingTime="217.907178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:39.832888485 +0000 UTC m=+573.689796240" watchObservedRunningTime="2026-10-30 00:44:39.833583434 +0000 UTC m=+573.690491190"
Oct 30 00:44:40.016217 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:40.016183    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:42.686894 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:42.686856    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:42.699293 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:42.699264    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:42.821422 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:42.821397    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb"
Oct 30 00:44:43.334302 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:43.334269    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:44:46.629314 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:44:46.629271    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:44:47.623319 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:47.623283    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:47.636733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:47.636696    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:47.853912 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:47.853873    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm"
Oct 30 00:44:49.016276 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:49.016244    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:49.029175 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:49.029143    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:49.742543 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:49.742457    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:44:49.860427 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:49.860400    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv"
Oct 30 00:44:57.677137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:57.677101    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68f8ff9ccb-jznqk"]
Oct 30 00:44:57.682299 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:57.682274    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68f8ff9ccb-jznqk"
Oct 30 00:44:57.698827 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:57.698800    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68f8ff9ccb-jznqk"]
Oct 30 00:44:57.760695 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:57.760666    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/78c5e37d-698d-492a-9870-f219f4d5beca-tls-cert\") pod \"authorino-68f8ff9ccb-jznqk\" (UID: \"78c5e37d-698d-492a-9870-f219f4d5beca\") " pod="kuadrant-system/authorino-68f8ff9ccb-jznqk"
Oct 30 00:44:57.760857 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:57.760733    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcgt2\" (UniqueName: \"kubernetes.io/projected/78c5e37d-698d-492a-9870-f219f4d5beca-kube-api-access-hcgt2\") pod \"authorino-68f8ff9ccb-jznqk\" (UID: \"78c5e37d-698d-492a-9870-f219f4d5beca\") " pod="kuadrant-system/authorino-68f8ff9ccb-jznqk"
Oct 30 00:44:57.861622 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:57.861595    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/78c5e37d-698d-492a-9870-f219f4d5beca-tls-cert\") pod \"authorino-68f8ff9ccb-jznqk\" (UID: \"78c5e37d-698d-492a-9870-f219f4d5beca\") " pod="kuadrant-system/authorino-68f8ff9ccb-jznqk"
Oct 30 00:44:57.861778 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:57.861646    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcgt2\" (UniqueName: \"kubernetes.io/projected/78c5e37d-698d-492a-9870-f219f4d5beca-kube-api-access-hcgt2\") pod \"authorino-68f8ff9ccb-jznqk\" (UID: \"78c5e37d-698d-492a-9870-f219f4d5beca\") " pod="kuadrant-system/authorino-68f8ff9ccb-jznqk"
Oct 30 00:44:57.863983 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:57.863958    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/78c5e37d-698d-492a-9870-f219f4d5beca-tls-cert\") pod \"authorino-68f8ff9ccb-jznqk\" (UID: \"78c5e37d-698d-492a-9870-f219f4d5beca\") " pod="kuadrant-system/authorino-68f8ff9ccb-jznqk"
Oct 30 00:44:57.882459 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:57.882436    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcgt2\" (UniqueName: \"kubernetes.io/projected/78c5e37d-698d-492a-9870-f219f4d5beca-kube-api-access-hcgt2\") pod \"authorino-68f8ff9ccb-jznqk\" (UID: \"78c5e37d-698d-492a-9870-f219f4d5beca\") " pod="kuadrant-system/authorino-68f8ff9ccb-jznqk"
Oct 30 00:44:57.997698 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:57.997636    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68f8ff9ccb-jznqk"
Oct 30 00:44:58.287527 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:58.287449    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68f8ff9ccb-jznqk"]
Oct 30 00:44:58.290976 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:44:58.290946    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78c5e37d_698d_492a_9870_f219f4d5beca.slice/crio-1737f5f7f2409903caa8bb94c2ccb6e1bbf37fb716c5d2ea96f9ae565e581f2c WatchSource:0}: Error finding container 1737f5f7f2409903caa8bb94c2ccb6e1bbf37fb716c5d2ea96f9ae565e581f2c: Status 404 returned error can't find the container with id 1737f5f7f2409903caa8bb94c2ccb6e1bbf37fb716c5d2ea96f9ae565e581f2c
Oct 30 00:44:58.892860 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:58.892826    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68f8ff9ccb-jznqk" event={"ID":"78c5e37d-698d-492a-9870-f219f4d5beca","Type":"ContainerStarted","Data":"1737f5f7f2409903caa8bb94c2ccb6e1bbf37fb716c5d2ea96f9ae565e581f2c"}
Oct 30 00:44:59.899851 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:59.899810    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68f8ff9ccb-jznqk" event={"ID":"78c5e37d-698d-492a-9870-f219f4d5beca","Type":"ContainerStarted","Data":"2a5ed56ad7f1c057df4d0780e15724a9f4204317f70b08942ab2c304f8c6581f"}
Oct 30 00:44:59.937230 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:59.937166    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-68f8ff9ccb-jznqk" podStartSLOduration=2.356885773 podStartE2EDuration="2.937146746s" totalImagesPullingTime="580.260973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:59.935970212 +0000 UTC m=+593.792877967" watchObservedRunningTime="2026-10-30 00:44:59.937146746 +0000 UTC m=+593.794054502"
Oct 30 00:44:59.964254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:59.964217    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56d59d5b8c-pf2rm"]
Oct 30 00:44:59.965094 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:44:59.964789    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56d59d5b8c-pf2rm" podUID="9e699c8a-dfe8-47d2-ba1f-8ebce73d7121" containerName="authorino" containerID="cri-o://92033c06cd82fb3d97ce81ea40be9f0e65fad9773e495be3ccfa77f353df1846" gracePeriod=30
Oct 30 00:45:00.150177 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.150100    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz"]
Oct 30 00:45:00.154464 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.154439    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz"
Oct 30 00:45:00.168354 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.168331    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:45:00.168354 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.168344    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:45:00.170559 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.170541    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:45:00.172332 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.172307    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz"]
Oct 30 00:45:00.243211 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.243183    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56d59d5b8c-pf2rm"
Oct 30 00:45:00.284634 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.284605    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac79ba0-cc4e-4848-8e8d-97722de763a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888685-bq4kz\" (UID: \"aac79ba0-cc4e-4848-8e8d-97722de763a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz"
Oct 30 00:45:00.284845 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.284820    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkq54\" (UniqueName: \"kubernetes.io/projected/aac79ba0-cc4e-4848-8e8d-97722de763a3-kube-api-access-nkq54\") pod \"maas-api-key-cleanup-29888685-bq4kz\" (UID: \"aac79ba0-cc4e-4848-8e8d-97722de763a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz"
Oct 30 00:45:00.386047 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.386010    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e699c8a-dfe8-47d2-ba1f-8ebce73d7121-kube-api-access-pk2bb\" (UniqueName: \"kubernetes.io/projected/9e699c8a-dfe8-47d2-ba1f-8ebce73d7121-kube-api-access-pk2bb\") pod \"9e699c8a-dfe8-47d2-ba1f-8ebce73d7121\" (UID: \"9e699c8a-dfe8-47d2-ba1f-8ebce73d7121\") "
Oct 30 00:45:00.386214 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.386074    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e699c8a-dfe8-47d2-ba1f-8ebce73d7121-tls-cert\" (UniqueName: \"kubernetes.io/secret/9e699c8a-dfe8-47d2-ba1f-8ebce73d7121-tls-cert\") pod \"9e699c8a-dfe8-47d2-ba1f-8ebce73d7121\" (UID: \"9e699c8a-dfe8-47d2-ba1f-8ebce73d7121\") "
Oct 30 00:45:00.386352 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.386332    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac79ba0-cc4e-4848-8e8d-97722de763a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888685-bq4kz\" (UID: \"aac79ba0-cc4e-4848-8e8d-97722de763a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz"
Oct 30 00:45:00.386547 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.386525    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkq54\" (UniqueName: \"kubernetes.io/projected/aac79ba0-cc4e-4848-8e8d-97722de763a3-kube-api-access-nkq54\") pod \"maas-api-key-cleanup-29888685-bq4kz\" (UID: \"aac79ba0-cc4e-4848-8e8d-97722de763a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz"
Oct 30 00:45:00.387142 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.387119    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac79ba0-cc4e-4848-8e8d-97722de763a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888685-bq4kz\" (UID: \"aac79ba0-cc4e-4848-8e8d-97722de763a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz"
Oct 30 00:45:00.388474 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.388445    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e699c8a-dfe8-47d2-ba1f-8ebce73d7121-kube-api-access-pk2bb" pod "9e699c8a-dfe8-47d2-ba1f-8ebce73d7121" (UID: "9e699c8a-dfe8-47d2-ba1f-8ebce73d7121"). InnerVolumeSpecName "kube-api-access-pk2bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:45:00.397636 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.397611    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e699c8a-dfe8-47d2-ba1f-8ebce73d7121-tls-cert" pod "9e699c8a-dfe8-47d2-ba1f-8ebce73d7121" (UID: "9e699c8a-dfe8-47d2-ba1f-8ebce73d7121"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:45:00.407553 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.407533    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkq54\" (UniqueName: \"kubernetes.io/projected/aac79ba0-cc4e-4848-8e8d-97722de763a3-kube-api-access-nkq54\") pod \"maas-api-key-cleanup-29888685-bq4kz\" (UID: \"aac79ba0-cc4e-4848-8e8d-97722de763a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz"
Oct 30 00:45:00.469553 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.469521    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz"
Oct 30 00:45:00.488082 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.488046    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk2bb\" (UniqueName: \"kubernetes.io/projected/9e699c8a-dfe8-47d2-ba1f-8ebce73d7121-kube-api-access-pk2bb\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:45:00.488225 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.488086    6099 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e699c8a-dfe8-47d2-ba1f-8ebce73d7121-tls-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:45:00.553836 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.551466    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-d8c8ccb-5r5q2"]
Oct 30 00:45:00.553836 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.551992    6099 cpu_manager.go:414] "containerMap: removing container" podUID="9e699c8a-dfe8-47d2-ba1f-8ebce73d7121" containerName="authorino"
Oct 30 00:45:00.553836 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.552008    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e699c8a-dfe8-47d2-ba1f-8ebce73d7121" containerName="authorino"
Oct 30 00:45:00.553836 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.552076    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e699c8a-dfe8-47d2-ba1f-8ebce73d7121" containerName="authorino"
Oct 30 00:45:00.555585 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.555560    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-d8c8ccb-5r5q2"
Oct 30 00:45:00.568382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.568359    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 00:45:00.568531 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.568496    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-wrqt2\""
Oct 30 00:45:00.568600 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.568545    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 00:45:00.575860 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.575808    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-d8c8ccb-5r5q2"]
Oct 30 00:45:00.690462 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.690382    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644a5fe7-0455-4a0e-a485-d4063388b732-metrics-certs\") pod \"ai-gateway-controller-d8c8ccb-5r5q2\" (UID: \"644a5fe7-0455-4a0e-a485-d4063388b732\") " pod="opendatahub/ai-gateway-controller-d8c8ccb-5r5q2"
Oct 30 00:45:00.690617 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.690467    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8dpp\" (UniqueName: \"kubernetes.io/projected/644a5fe7-0455-4a0e-a485-d4063388b732-kube-api-access-l8dpp\") pod \"ai-gateway-controller-d8c8ccb-5r5q2\" (UID: \"644a5fe7-0455-4a0e-a485-d4063388b732\") " pod="opendatahub/ai-gateway-controller-d8c8ccb-5r5q2"
Oct 30 00:45:00.791759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.791694    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644a5fe7-0455-4a0e-a485-d4063388b732-metrics-certs\") pod \"ai-gateway-controller-d8c8ccb-5r5q2\" (UID: \"644a5fe7-0455-4a0e-a485-d4063388b732\") " pod="opendatahub/ai-gateway-controller-d8c8ccb-5r5q2"
Oct 30 00:45:00.791931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.791785    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8dpp\" (UniqueName: \"kubernetes.io/projected/644a5fe7-0455-4a0e-a485-d4063388b732-kube-api-access-l8dpp\") pod \"ai-gateway-controller-d8c8ccb-5r5q2\" (UID: \"644a5fe7-0455-4a0e-a485-d4063388b732\") " pod="opendatahub/ai-gateway-controller-d8c8ccb-5r5q2"
Oct 30 00:45:00.791931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.791817    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz"]
Oct 30 00:45:00.794647 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.794621    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644a5fe7-0455-4a0e-a485-d4063388b732-metrics-certs\") pod \"ai-gateway-controller-d8c8ccb-5r5q2\" (UID: \"644a5fe7-0455-4a0e-a485-d4063388b732\") " pod="opendatahub/ai-gateway-controller-d8c8ccb-5r5q2"
Oct 30 00:45:00.795886 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:45:00.795862    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac79ba0_cc4e_4848_8e8d_97722de763a3.slice/crio-f298577f201e4cf38032ca0595aac60ddc129de67796ed67871f0132e55e8104 WatchSource:0}: Error finding container f298577f201e4cf38032ca0595aac60ddc129de67796ed67871f0132e55e8104: Status 404 returned error can't find the container with id f298577f201e4cf38032ca0595aac60ddc129de67796ed67871f0132e55e8104
Oct 30 00:45:00.813411 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.813378    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8dpp\" (UniqueName: \"kubernetes.io/projected/644a5fe7-0455-4a0e-a485-d4063388b732-kube-api-access-l8dpp\") pod \"ai-gateway-controller-d8c8ccb-5r5q2\" (UID: \"644a5fe7-0455-4a0e-a485-d4063388b732\") " pod="opendatahub/ai-gateway-controller-d8c8ccb-5r5q2"
Oct 30 00:45:00.874158 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.874130    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-d8c8ccb-5r5q2"
Oct 30 00:45:00.906059 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.906028    6099 generic.go:356] "Generic (PLEG): container finished" podID="9e699c8a-dfe8-47d2-ba1f-8ebce73d7121" containerID="92033c06cd82fb3d97ce81ea40be9f0e65fad9773e495be3ccfa77f353df1846" exitCode=0
Oct 30 00:45:00.906495 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.906117    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56d59d5b8c-pf2rm"
Oct 30 00:45:00.906495 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.906109    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56d59d5b8c-pf2rm" event={"ID":"9e699c8a-dfe8-47d2-ba1f-8ebce73d7121","Type":"ContainerDied","Data":"92033c06cd82fb3d97ce81ea40be9f0e65fad9773e495be3ccfa77f353df1846"}
Oct 30 00:45:00.906495 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.906269    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56d59d5b8c-pf2rm" event={"ID":"9e699c8a-dfe8-47d2-ba1f-8ebce73d7121","Type":"ContainerDied","Data":"fa9e4773b33072ddb39dc9f74f93799f96b83b089a25b8f06dac86f50c8df3af"}
Oct 30 00:45:00.906495 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.906297    6099 scope.go:122] "RemoveContainer" containerID="92033c06cd82fb3d97ce81ea40be9f0e65fad9773e495be3ccfa77f353df1846"
Oct 30 00:45:00.907795 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.907599    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz" event={"ID":"aac79ba0-cc4e-4848-8e8d-97722de763a3","Type":"ContainerStarted","Data":"f298577f201e4cf38032ca0595aac60ddc129de67796ed67871f0132e55e8104"}
Oct 30 00:45:00.917455 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.915475    6099 scope.go:122] "RemoveContainer" containerID="92033c06cd82fb3d97ce81ea40be9f0e65fad9773e495be3ccfa77f353df1846"
Oct 30 00:45:00.917455 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:45:00.915829    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92033c06cd82fb3d97ce81ea40be9f0e65fad9773e495be3ccfa77f353df1846\": container with ID starting with 92033c06cd82fb3d97ce81ea40be9f0e65fad9773e495be3ccfa77f353df1846 not found: ID does not exist" containerID="92033c06cd82fb3d97ce81ea40be9f0e65fad9773e495be3ccfa77f353df1846"
Oct 30 00:45:00.917455 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.915874    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92033c06cd82fb3d97ce81ea40be9f0e65fad9773e495be3ccfa77f353df1846"} err="failed to get container status \"92033c06cd82fb3d97ce81ea40be9f0e65fad9773e495be3ccfa77f353df1846\": rpc error: code = NotFound desc = could not find container \"92033c06cd82fb3d97ce81ea40be9f0e65fad9773e495be3ccfa77f353df1846\": container with ID starting with 92033c06cd82fb3d97ce81ea40be9f0e65fad9773e495be3ccfa77f353df1846 not found: ID does not exist"
Oct 30 00:45:00.953749 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.953311    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56d59d5b8c-pf2rm"]
Oct 30 00:45:00.956900 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:00.956842    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56d59d5b8c-pf2rm"]
Oct 30 00:45:01.172639 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:01.172583    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-d8c8ccb-5r5q2"]
Oct 30 00:45:01.176181 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:45:01.176137    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644a5fe7_0455_4a0e_a485_d4063388b732.slice/crio-35fa1d0e7d7fbe04eb5047a1d4018724f2cda274db0e08aa19e68eb965dcba26 WatchSource:0}: Error finding container 35fa1d0e7d7fbe04eb5047a1d4018724f2cda274db0e08aa19e68eb965dcba26: Status 404 returned error can't find the container with id 35fa1d0e7d7fbe04eb5047a1d4018724f2cda274db0e08aa19e68eb965dcba26
Oct 30 00:45:01.920102 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:01.920061    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-d8c8ccb-5r5q2" event={"ID":"644a5fe7-0455-4a0e-a485-d4063388b732","Type":"ContainerStarted","Data":"35fa1d0e7d7fbe04eb5047a1d4018724f2cda274db0e08aa19e68eb965dcba26"}
Oct 30 00:45:02.263249 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:02.263168    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e699c8a-dfe8-47d2-ba1f-8ebce73d7121" path="/var/lib/kubelet/pods/9e699c8a-dfe8-47d2-ba1f-8ebce73d7121/volumes"
Oct 30 00:45:04.939492 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:04.939408    6099 generic.go:356] "Generic (PLEG): container finished" podID="aac79ba0-cc4e-4848-8e8d-97722de763a3" containerID="bc5e86b614124d606a320d57c3f18ae93799ac250e071221624f628567b56eff" exitCode=60
Oct 30 00:45:04.939988 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:04.939499    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz" event={"ID":"aac79ba0-cc4e-4848-8e8d-97722de763a3","Type":"ContainerDied","Data":"bc5e86b614124d606a320d57c3f18ae93799ac250e071221624f628567b56eff"}
Oct 30 00:45:04.939988 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:04.939759    6099 scope.go:122] "RemoveContainer" containerID="bc5e86b614124d606a320d57c3f18ae93799ac250e071221624f628567b56eff"
Oct 30 00:45:04.941396 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:04.941372    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-d8c8ccb-5r5q2" event={"ID":"644a5fe7-0455-4a0e-a485-d4063388b732","Type":"ContainerStarted","Data":"724d19ae3c3678d60d2447f0a1924d7a618ddd22af5af36cc6f8824f1e67b649"}
Oct 30 00:45:04.941525 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:04.941511    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-d8c8ccb-5r5q2"
Oct 30 00:45:05.013026 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:05.012974    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-d8c8ccb-5r5q2" podStartSLOduration=2.054114324 podStartE2EDuration="5.012960577s" totalImagesPullingTime="2.958846253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:05.012465187 +0000 UTC m=+598.869372933" watchObservedRunningTime="2026-10-30 00:45:05.012960577 +0000 UTC m=+598.869868343"
Oct 30 00:45:05.741434 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:05.741386    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-rcxc5"]
Oct 30 00:45:05.745354 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:05.745334    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-rcxc5"
Oct 30 00:45:05.760313 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:05.760291    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1569a420-ef8b-4ca9-86c1-cde78845b4b6-plugins-config-volume\") pod \"payload-processing-5f78687d5b-rcxc5\" (UID: \"1569a420-ef8b-4ca9-86c1-cde78845b4b6\") " pod="openshift-ingress/payload-processing-5f78687d5b-rcxc5"
Oct 30 00:45:05.760464 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:05.760446    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7vfg\" (UniqueName: \"kubernetes.io/projected/1569a420-ef8b-4ca9-86c1-cde78845b4b6-kube-api-access-m7vfg\") pod \"payload-processing-5f78687d5b-rcxc5\" (UID: \"1569a420-ef8b-4ca9-86c1-cde78845b4b6\") " pod="openshift-ingress/payload-processing-5f78687d5b-rcxc5"
Oct 30 00:45:05.762747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:05.762701    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2pcqk\""
Oct 30 00:45:05.771546 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:05.771519    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-rcxc5"]
Oct 30 00:45:05.790203 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:05.790183    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 00:45:05.861751 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:05.861697    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7vfg\" (UniqueName: \"kubernetes.io/projected/1569a420-ef8b-4ca9-86c1-cde78845b4b6-kube-api-access-m7vfg\") pod \"payload-processing-5f78687d5b-rcxc5\" (UID: \"1569a420-ef8b-4ca9-86c1-cde78845b4b6\") " pod="openshift-ingress/payload-processing-5f78687d5b-rcxc5"
Oct 30 00:45:05.861886 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:05.861803    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1569a420-ef8b-4ca9-86c1-cde78845b4b6-plugins-config-volume\") pod \"payload-processing-5f78687d5b-rcxc5\" (UID: \"1569a420-ef8b-4ca9-86c1-cde78845b4b6\") " pod="openshift-ingress/payload-processing-5f78687d5b-rcxc5"
Oct 30 00:45:05.862451 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:05.862427    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1569a420-ef8b-4ca9-86c1-cde78845b4b6-plugins-config-volume\") pod \"payload-processing-5f78687d5b-rcxc5\" (UID: \"1569a420-ef8b-4ca9-86c1-cde78845b4b6\") " pod="openshift-ingress/payload-processing-5f78687d5b-rcxc5"
Oct 30 00:45:05.882568 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:05.882537    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7vfg\" (UniqueName: \"kubernetes.io/projected/1569a420-ef8b-4ca9-86c1-cde78845b4b6-kube-api-access-m7vfg\") pod \"payload-processing-5f78687d5b-rcxc5\" (UID: \"1569a420-ef8b-4ca9-86c1-cde78845b4b6\") " pod="openshift-ingress/payload-processing-5f78687d5b-rcxc5"
Oct 30 00:45:05.954082 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:05.954046    6099 generic.go:356] "Generic (PLEG): container finished" podID="aac79ba0-cc4e-4848-8e8d-97722de763a3" containerID="e7af400433cea4f51e42a985970a8a041a4b03389fcfdb4197bac2c6c46d78c1" exitCode=60
Oct 30 00:45:05.954492 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:05.954127    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz" event={"ID":"aac79ba0-cc4e-4848-8e8d-97722de763a3","Type":"ContainerDied","Data":"e7af400433cea4f51e42a985970a8a041a4b03389fcfdb4197bac2c6c46d78c1"}
Oct 30 00:45:05.954492 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:05.954218    6099 scope.go:122] "RemoveContainer" containerID="bc5e86b614124d606a320d57c3f18ae93799ac250e071221624f628567b56eff"
Oct 30 00:45:05.954492 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:05.954447    6099 scope.go:122] "RemoveContainer" containerID="e7af400433cea4f51e42a985970a8a041a4b03389fcfdb4197bac2c6c46d78c1"
Oct 30 00:45:05.954684 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:45:05.954665    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888685-bq4kz_odh-ai-gateway-infra(aac79ba0-cc4e-4848-8e8d-97722de763a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz" podUID="aac79ba0-cc4e-4848-8e8d-97722de763a3"
Oct 30 00:45:06.057945 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.057920    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-rcxc5"
Oct 30 00:45:06.100735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.099181    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"]
Oct 30 00:45:06.104134 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.104106    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.120116 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.119958    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 00:45:06.120116 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.120085    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-76qjj\""
Oct 30 00:45:06.130549 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.130491    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"]
Oct 30 00:45:06.163805 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.163778    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.163918 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.163816    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plnvp\" (UniqueName: \"kubernetes.io/projected/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-kube-api-access-plnvp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.163918 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.163883    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.163918 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.163904    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.164028 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.163965    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.164028 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.163998    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.164028 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.164021    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.164150 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.164132    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.164184 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.164170    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.265335 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.265306    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.265335 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.265338    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.265567 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.265366    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.265567 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.265407    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.265567 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.265457    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.265567 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.265516    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.265567 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.265562    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.265841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.265597    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.265841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.265632    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plnvp\" (UniqueName: \"kubernetes.io/projected/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-kube-api-access-plnvp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.265841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.265770    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.265841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.265801    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.265841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.265816    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.266064 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.266046    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.267635 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.267610    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.267855 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.267834    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.277064 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.277043    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 00:45:06.286391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.286366    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.286510 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.286497    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.286573 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.286556    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plnvp\" (UniqueName: \"kubernetes.io/projected/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-kube-api-access-plnvp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.323856 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.323833    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 00:45:06.327461 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.327441    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 00:45:06.355728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.355685    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-rcxc5"]
Oct 30 00:45:06.358601 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:45:06.358580    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1569a420_ef8b_4ca9_86c1_cde78845b4b6.slice/crio-6a7ed1c4263229787b0cb32dd6091cb8afcf9c9186f1f439964d796b61c5c0c7 WatchSource:0}: Error finding container 6a7ed1c4263229787b0cb32dd6091cb8afcf9c9186f1f439964d796b61c5c0c7: Status 404 returned error can't find the container with id 6a7ed1c4263229787b0cb32dd6091cb8afcf9c9186f1f439964d796b61c5c0c7
Oct 30 00:45:06.434810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.434783    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-76qjj\""
Oct 30 00:45:06.443609 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.443583    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:06.752678 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.752642    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"]
Oct 30 00:45:06.755219 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:45:06.755191    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f57545_b25c_45ac_9fd5_bc9c6e1cf068.slice/crio-a9b159cc7b94d608fe777aaed85758c396433af580fe9802708e6a23a84f7f49 WatchSource:0}: Error finding container a9b159cc7b94d608fe777aaed85758c396433af580fe9802708e6a23a84f7f49: Status 404 returned error can't find the container with id a9b159cc7b94d608fe777aaed85758c396433af580fe9802708e6a23a84f7f49
Oct 30 00:45:06.962580 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.962551    6099 scope.go:122] "RemoveContainer" containerID="e7af400433cea4f51e42a985970a8a041a4b03389fcfdb4197bac2c6c46d78c1"
Oct 30 00:45:06.963024 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:45:06.962907    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888685-bq4kz_odh-ai-gateway-infra(aac79ba0-cc4e-4848-8e8d-97722de763a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz" podUID="aac79ba0-cc4e-4848-8e8d-97722de763a3"
Oct 30 00:45:06.964797 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.964736    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm" event={"ID":"04f57545-b25c-45ac-9fd5-bc9c6e1cf068","Type":"ContainerStarted","Data":"a9b159cc7b94d608fe777aaed85758c396433af580fe9802708e6a23a84f7f49"}
Oct 30 00:45:06.966222 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:06.966188    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-rcxc5" event={"ID":"1569a420-ef8b-4ca9-86c1-cde78845b4b6","Type":"ContainerStarted","Data":"6a7ed1c4263229787b0cb32dd6091cb8afcf9c9186f1f439964d796b61c5c0c7"}
Oct 30 00:45:08.977471 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:08.977433    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-rcxc5" event={"ID":"1569a420-ef8b-4ca9-86c1-cde78845b4b6","Type":"ContainerStarted","Data":"4f0d052573ddf3c7272091fbbf6613e63e3d6dff211da987506e123da06ba59c"}
Oct 30 00:45:08.977888 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:08.977569    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-rcxc5"
Oct 30 00:45:09.016990 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:09.016298    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-rcxc5" podStartSLOduration=1.89750485 podStartE2EDuration="4.016280657s" totalImagesPullingTime="2.118775807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:09.015841335 +0000 UTC m=+602.872749089" watchObservedRunningTime="2026-10-30 00:45:09.016280657 +0000 UTC m=+602.873188413"
Oct 30 00:45:10.214349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:10.214315    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:45:10.214589 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:10.214387    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:45:10.214589 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:10.214416    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:45:10.995531 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:10.995495    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm" event={"ID":"04f57545-b25c-45ac-9fd5-bc9c6e1cf068","Type":"ContainerStarted","Data":"b802d4d73dd948541c74cd0ca0ba2290c3c4fa2efe1b1f80de14208ac2da582e"}
Oct 30 00:45:11.051294 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:11.051243    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm" podStartSLOduration=1.595004285 podStartE2EDuration="5.05122692s" totalImagesPullingTime="3.456222635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:11.050903711 +0000 UTC m=+604.907811468" watchObservedRunningTime="2026-10-30 00:45:11.05122692 +0000 UTC m=+604.908134675"
Oct 30 00:45:11.444440 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:11.444401    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:11.446013 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:11.445986    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:45:11.446138 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:11.446035    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm" podUID="04f57545-b25c-45ac-9fd5-bc9c6e1cf068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 00:45:12.444178 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:12.444140    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:45:12.444348 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:12.444194    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm" podUID="04f57545-b25c-45ac-9fd5-bc9c6e1cf068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 00:45:13.444488 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:13.444445    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:45:13.444968 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:13.444507    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm" podUID="04f57545-b25c-45ac-9fd5-bc9c6e1cf068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 00:45:14.444264 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:14.444228    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:45:14.444457 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:14.444290    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm" podUID="04f57545-b25c-45ac-9fd5-bc9c6e1cf068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 00:45:15.444456 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:15.444422    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:45:15.444839 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:15.444474    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm" podUID="04f57545-b25c-45ac-9fd5-bc9c6e1cf068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 00:45:15.956990 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:15.956958    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-d8c8ccb-5r5q2"
Oct 30 00:45:16.245124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:16.245028    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-84479857d9-2xt9z"]
Oct 30 00:45:16.249236 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:16.249210    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84479857d9-2xt9z"
Oct 30 00:45:16.269748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:16.269722    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-84479857d9-2xt9z"]
Oct 30 00:45:16.396022 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:16.395993    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c6a2a03-fda8-46b2-9246-4d77746c9421-metrics-certs\") pod \"ai-gateway-controller-84479857d9-2xt9z\" (UID: \"5c6a2a03-fda8-46b2-9246-4d77746c9421\") " pod="opendatahub/ai-gateway-controller-84479857d9-2xt9z"
Oct 30 00:45:16.396022 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:16.396022    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbdvd\" (UniqueName: \"kubernetes.io/projected/5c6a2a03-fda8-46b2-9246-4d77746c9421-kube-api-access-vbdvd\") pod \"ai-gateway-controller-84479857d9-2xt9z\" (UID: \"5c6a2a03-fda8-46b2-9246-4d77746c9421\") " pod="opendatahub/ai-gateway-controller-84479857d9-2xt9z"
Oct 30 00:45:16.444137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:16.444113    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:45:16.444209 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:16.444167    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm" podUID="04f57545-b25c-45ac-9fd5-bc9c6e1cf068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 00:45:16.497249 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:16.497195    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c6a2a03-fda8-46b2-9246-4d77746c9421-metrics-certs\") pod \"ai-gateway-controller-84479857d9-2xt9z\" (UID: \"5c6a2a03-fda8-46b2-9246-4d77746c9421\") " pod="opendatahub/ai-gateway-controller-84479857d9-2xt9z"
Oct 30 00:45:16.497249 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:16.497225    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbdvd\" (UniqueName: \"kubernetes.io/projected/5c6a2a03-fda8-46b2-9246-4d77746c9421-kube-api-access-vbdvd\") pod \"ai-gateway-controller-84479857d9-2xt9z\" (UID: \"5c6a2a03-fda8-46b2-9246-4d77746c9421\") " pod="opendatahub/ai-gateway-controller-84479857d9-2xt9z"
Oct 30 00:45:16.499587 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:16.499561    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c6a2a03-fda8-46b2-9246-4d77746c9421-metrics-certs\") pod \"ai-gateway-controller-84479857d9-2xt9z\" (UID: \"5c6a2a03-fda8-46b2-9246-4d77746c9421\") " pod="opendatahub/ai-gateway-controller-84479857d9-2xt9z"
Oct 30 00:45:16.518971 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:16.518947    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbdvd\" (UniqueName: \"kubernetes.io/projected/5c6a2a03-fda8-46b2-9246-4d77746c9421-kube-api-access-vbdvd\") pod \"ai-gateway-controller-84479857d9-2xt9z\" (UID: \"5c6a2a03-fda8-46b2-9246-4d77746c9421\") " pod="opendatahub/ai-gateway-controller-84479857d9-2xt9z"
Oct 30 00:45:16.562897 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:16.562868    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84479857d9-2xt9z"
Oct 30 00:45:16.638488 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:45:16.638453    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:45:16.857866 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:16.857837    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-84479857d9-2xt9z"]
Oct 30 00:45:16.861607 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:45:16.861584    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6a2a03_fda8_46b2_9246_4d77746c9421.slice/crio-0b2786714c585afef94d697d8f31f7ba0fcd104ae19f8d9672269ed07b12a765 WatchSource:0}: Error finding container 0b2786714c585afef94d697d8f31f7ba0fcd104ae19f8d9672269ed07b12a765: Status 404 returned error can't find the container with id 0b2786714c585afef94d697d8f31f7ba0fcd104ae19f8d9672269ed07b12a765
Oct 30 00:45:17.023776 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:17.023741    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84479857d9-2xt9z" event={"ID":"5c6a2a03-fda8-46b2-9246-4d77746c9421","Type":"ContainerStarted","Data":"0b2786714c585afef94d697d8f31f7ba0fcd104ae19f8d9672269ed07b12a765"}
Oct 30 00:45:17.254348 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:17.254327    6099 scope.go:122] "RemoveContainer" containerID="e7af400433cea4f51e42a985970a8a041a4b03389fcfdb4197bac2c6c46d78c1"
Oct 30 00:45:17.444197 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:17.444159    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:45:17.444332 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:17.444214    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm" podUID="04f57545-b25c-45ac-9fd5-bc9c6e1cf068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 00:45:18.040174 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:18.040126    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz" event={"ID":"aac79ba0-cc4e-4848-8e8d-97722de763a3","Type":"ContainerStarted","Data":"b796f3924ff3b2d25c4ce64490f04786d5461831c27622fb8e5cb33f417ccba6"}
Oct 30 00:45:18.041751 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:18.041698    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84479857d9-2xt9z" event={"ID":"5c6a2a03-fda8-46b2-9246-4d77746c9421","Type":"ContainerStarted","Data":"7748c2d850cd62d66fcf8b50833a9d51320108ebf2064676d6746e84cbb184c1"}
Oct 30 00:45:18.041886 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:18.041771    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-84479857d9-2xt9z"
Oct 30 00:45:18.080895 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:18.080844    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz" podStartSLOduration=15.14722991 podStartE2EDuration="18.080829371s" totalImagesPullingTime="2.933599461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:18.079793947 +0000 UTC m=+611.936701703" watchObservedRunningTime="2026-10-30 00:45:18.080829371 +0000 UTC m=+611.937737131"
Oct 30 00:45:18.117291 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:18.117248    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-84479857d9-2xt9z" podStartSLOduration=1.769810672 podStartE2EDuration="2.117235982s" totalImagesPullingTime="347.42531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:18.11651275 +0000 UTC m=+611.973420507" watchObservedRunningTime="2026-10-30 00:45:18.117235982 +0000 UTC m=+611.974143737"
Oct 30 00:45:18.613520 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:18.613490    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:18.614185 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:18.614081    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:18.614831 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:18.614809    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 00:45:19.047827 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:19.047797    6099 generic.go:356] "Generic (PLEG): container finished" podID="aac79ba0-cc4e-4848-8e8d-97722de763a3" containerID="b796f3924ff3b2d25c4ce64490f04786d5461831c27622fb8e5cb33f417ccba6" exitCode=60
Oct 30 00:45:19.048297 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:19.047871    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz" event={"ID":"aac79ba0-cc4e-4848-8e8d-97722de763a3","Type":"ContainerDied","Data":"b796f3924ff3b2d25c4ce64490f04786d5461831c27622fb8e5cb33f417ccba6"}
Oct 30 00:45:19.048297 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:19.047944    6099 scope.go:122] "RemoveContainer" containerID="e7af400433cea4f51e42a985970a8a041a4b03389fcfdb4197bac2c6c46d78c1"
Oct 30 00:45:19.048460 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:19.048384    6099 scope.go:122] "RemoveContainer" containerID="b796f3924ff3b2d25c4ce64490f04786d5461831c27622fb8e5cb33f417ccba6"
Oct 30 00:45:19.048589 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:45:19.048565    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888685-bq4kz_odh-ai-gateway-infra(aac79ba0-cc4e-4848-8e8d-97722de763a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz" podUID="aac79ba0-cc4e-4848-8e8d-97722de763a3"
Oct 30 00:45:19.095476 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:19.095443    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz"]
Oct 30 00:45:19.986185 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:19.986151    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-rcxc5"
Oct 30 00:45:20.189388 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:20.189365    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz"
Oct 30 00:45:20.335157 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:20.335130    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac79ba0-cc4e-4848-8e8d-97722de763a3-kube-api-access-nkq54\" (UniqueName: \"kubernetes.io/projected/aac79ba0-cc4e-4848-8e8d-97722de763a3-kube-api-access-nkq54\") pod \"aac79ba0-cc4e-4848-8e8d-97722de763a3\" (UID: \"aac79ba0-cc4e-4848-8e8d-97722de763a3\") "
Oct 30 00:45:20.335309 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:20.335218    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aac79ba0-cc4e-4848-8e8d-97722de763a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac79ba0-cc4e-4848-8e8d-97722de763a3-maas-api-service-ca\") pod \"aac79ba0-cc4e-4848-8e8d-97722de763a3\" (UID: \"aac79ba0-cc4e-4848-8e8d-97722de763a3\") "
Oct 30 00:45:20.335545 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:20.335518    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac79ba0-cc4e-4848-8e8d-97722de763a3-maas-api-service-ca" pod "aac79ba0-cc4e-4848-8e8d-97722de763a3" (UID: "aac79ba0-cc4e-4848-8e8d-97722de763a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:45:20.337305 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:20.337286    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac79ba0-cc4e-4848-8e8d-97722de763a3-kube-api-access-nkq54" pod "aac79ba0-cc4e-4848-8e8d-97722de763a3" (UID: "aac79ba0-cc4e-4848-8e8d-97722de763a3"). InnerVolumeSpecName "kube-api-access-nkq54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:45:20.436067 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:20.436040    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac79ba0-cc4e-4848-8e8d-97722de763a3-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:45:20.436067 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:20.436062    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkq54\" (UniqueName: \"kubernetes.io/projected/aac79ba0-cc4e-4848-8e8d-97722de763a3-kube-api-access-nkq54\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:45:21.060819 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.060784    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz" event={"ID":"aac79ba0-cc4e-4848-8e8d-97722de763a3","Type":"ContainerDied","Data":"f298577f201e4cf38032ca0595aac60ddc129de67796ed67871f0132e55e8104"}
Oct 30 00:45:21.060819 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.060815    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz"
Oct 30 00:45:21.061057 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.060847    6099 scope.go:122] "RemoveContainer" containerID="b796f3924ff3b2d25c4ce64490f04786d5461831c27622fb8e5cb33f417ccba6"
Oct 30 00:45:21.142469 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.142434    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz"]
Oct 30 00:45:21.152548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.152526    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888685-bq4kz"]
Oct 30 00:45:21.174246 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.174223    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"]
Oct 30 00:45:21.174786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.174764    6099 cpu_manager.go:414] "containerMap: removing container" podUID="aac79ba0-cc4e-4848-8e8d-97722de763a3" containerName="cleanup"
Oct 30 00:45:21.174786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.174786    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac79ba0-cc4e-4848-8e8d-97722de763a3" containerName="cleanup"
Oct 30 00:45:21.174921 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.174796    6099 cpu_manager.go:414] "containerMap: removing container" podUID="aac79ba0-cc4e-4848-8e8d-97722de763a3" containerName="cleanup"
Oct 30 00:45:21.174921 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.174803    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac79ba0-cc4e-4848-8e8d-97722de763a3" containerName="cleanup"
Oct 30 00:45:21.174921 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.174885    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac79ba0-cc4e-4848-8e8d-97722de763a3" containerName="cleanup"
Oct 30 00:45:21.174921 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.174893    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac79ba0-cc4e-4848-8e8d-97722de763a3" containerName="cleanup"
Oct 30 00:45:21.174921 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.174904    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac79ba0-cc4e-4848-8e8d-97722de763a3" containerName="cleanup"
Oct 30 00:45:21.179052 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.179034    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.203923 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.203900    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"]
Oct 30 00:45:21.344312 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.344228    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.344312 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.344263    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.344312 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.344289    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.344577 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.344341    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.344577 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.344480    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.344657 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.344587    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.344657 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.344607    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.344657 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.344631    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb2mg\" (UniqueName: \"kubernetes.io/projected/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-kube-api-access-mb2mg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.344657 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.344652    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.445221 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.445187    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.445363 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.445251    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.445363 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.445269    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.445363 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.445294    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.445363 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.445310    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.445363 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.445330    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.445637 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.445387    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.445637 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.445411    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.445637 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.445437    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb2mg\" (UniqueName: \"kubernetes.io/projected/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-kube-api-access-mb2mg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.445896 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.445674    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.445896 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.445845    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.445896 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.445875    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.446068 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.446045    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.446276 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.446260    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.447605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.447588    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.447855 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.447837    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.466489 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.466462    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.466587 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.466550    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb2mg\" (UniqueName: \"kubernetes.io/projected/1f88f534-38ba-41ff-b2e5-b7335f2f1fb2-kube-api-access-mb2mg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7nffr\" (UID: \"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.494489 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.494467    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:21.823113 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.821936    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"]
Oct 30 00:45:21.826406 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:45:21.826376    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f88f534_38ba_41ff_b2e5_b7335f2f1fb2.slice/crio-2208b9da1c81c2625a3bcb630a57d46a9cc0330cd9d96d6ded35eadac359e070 WatchSource:0}: Error finding container 2208b9da1c81c2625a3bcb630a57d46a9cc0330cd9d96d6ded35eadac359e070: Status 404 returned error can't find the container with id 2208b9da1c81c2625a3bcb630a57d46a9cc0330cd9d96d6ded35eadac359e070
Oct 30 00:45:21.829472 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.829436    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:45:21.829549 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.829519    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:45:21.829589 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:21.829547    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:45:22.069458 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:22.069421    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr" event={"ID":"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2","Type":"ContainerStarted","Data":"eb24239cad1d937cbb29a91ec80cde0df7cd201ab1b57781dcbf47bda078e8df"}
Oct 30 00:45:22.069649 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:22.069478    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr" event={"ID":"1f88f534-38ba-41ff-b2e5-b7335f2f1fb2","Type":"ContainerStarted","Data":"2208b9da1c81c2625a3bcb630a57d46a9cc0330cd9d96d6ded35eadac359e070"}
Oct 30 00:45:22.113190 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:22.113083    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr" podStartSLOduration=1.113065463 podStartE2EDuration="1.113065463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:22.110800471 +0000 UTC m=+615.967708227" watchObservedRunningTime="2026-10-30 00:45:22.113065463 +0000 UTC m=+615.969973216"
Oct 30 00:45:22.259837 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:22.259804    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aac79ba0-cc4e-4848-8e8d-97722de763a3" path="/var/lib/kubelet/pods/aac79ba0-cc4e-4848-8e8d-97722de763a3/volumes"
Oct 30 00:45:22.495185 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:22.495093    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:22.496652 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:22.496624    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 00:45:22.496788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:22.496678    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr" podUID="1f88f534-38ba-41ff-b2e5-b7335f2f1fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 00:45:23.495170 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:23.495134    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 00:45:23.495550 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:23.495185    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr" podUID="1f88f534-38ba-41ff-b2e5-b7335f2f1fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 00:45:24.495389 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:24.495354    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 00:45:24.495776 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:24.495412    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr" podUID="1f88f534-38ba-41ff-b2e5-b7335f2f1fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 00:45:25.495716 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:25.495680    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 00:45:25.496090 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:25.495753    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr" podUID="1f88f534-38ba-41ff-b2e5-b7335f2f1fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 00:45:26.494944 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:26.494910    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 00:45:26.495117 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:26.494978    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr" podUID="1f88f534-38ba-41ff-b2e5-b7335f2f1fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 00:45:27.495222 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:27.495183    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 00:45:27.495635 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:27.495251    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr" podUID="1f88f534-38ba-41ff-b2e5-b7335f2f1fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 00:45:28.495582 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:28.495541    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 00:45:28.495978 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:28.495621    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr" podUID="1f88f534-38ba-41ff-b2e5-b7335f2f1fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 00:45:29.050617 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:29.050586    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-84479857d9-2xt9z"
Oct 30 00:45:29.145123 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:29.145089    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-d8c8ccb-5r5q2"]
Oct 30 00:45:29.145377 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:29.145348    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-d8c8ccb-5r5q2" podUID="644a5fe7-0455-4a0e-a485-d4063388b732" containerName="manager" containerID="cri-o://724d19ae3c3678d60d2447f0a1924d7a618ddd22af5af36cc6f8824f1e67b649" gracePeriod=30
Oct 30 00:45:29.431558 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:29.431523    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-d8c8ccb-5r5q2"
Oct 30 00:45:29.453330 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:29.453306    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/644a5fe7-0455-4a0e-a485-d4063388b732-kube-api-access-l8dpp\" (UniqueName: \"kubernetes.io/projected/644a5fe7-0455-4a0e-a485-d4063388b732-kube-api-access-l8dpp\") pod \"644a5fe7-0455-4a0e-a485-d4063388b732\" (UID: \"644a5fe7-0455-4a0e-a485-d4063388b732\") "
Oct 30 00:45:29.453481 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:29.453367    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/644a5fe7-0455-4a0e-a485-d4063388b732-metrics-certs\" (UniqueName: \"kubernetes.io/secret/644a5fe7-0455-4a0e-a485-d4063388b732-metrics-certs\") pod \"644a5fe7-0455-4a0e-a485-d4063388b732\" (UID: \"644a5fe7-0455-4a0e-a485-d4063388b732\") "
Oct 30 00:45:29.455470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:29.455445    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644a5fe7-0455-4a0e-a485-d4063388b732-kube-api-access-l8dpp" pod "644a5fe7-0455-4a0e-a485-d4063388b732" (UID: "644a5fe7-0455-4a0e-a485-d4063388b732"). InnerVolumeSpecName "kube-api-access-l8dpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:45:29.455569 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:29.455480    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644a5fe7-0455-4a0e-a485-d4063388b732-metrics-certs" pod "644a5fe7-0455-4a0e-a485-d4063388b732" (UID: "644a5fe7-0455-4a0e-a485-d4063388b732"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:45:29.554912 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:29.554875    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8dpp\" (UniqueName: \"kubernetes.io/projected/644a5fe7-0455-4a0e-a485-d4063388b732-kube-api-access-l8dpp\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:45:29.554912 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:29.554901    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644a5fe7-0455-4a0e-a485-d4063388b732-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:45:30.105175 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:30.105143    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:30.105542 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:30.105515    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:30.106250 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:30.106229    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr"
Oct 30 00:45:30.127357 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:30.127329    6099 generic.go:356] "Generic (PLEG): container finished" podID="644a5fe7-0455-4a0e-a485-d4063388b732" containerID="724d19ae3c3678d60d2447f0a1924d7a618ddd22af5af36cc6f8824f1e67b649" exitCode=0
Oct 30 00:45:30.127480 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:30.127372    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-d8c8ccb-5r5q2"
Oct 30 00:45:30.127480 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:30.127403    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-d8c8ccb-5r5q2" event={"ID":"644a5fe7-0455-4a0e-a485-d4063388b732","Type":"ContainerDied","Data":"724d19ae3c3678d60d2447f0a1924d7a618ddd22af5af36cc6f8824f1e67b649"}
Oct 30 00:45:30.127480 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:30.127452    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-d8c8ccb-5r5q2" event={"ID":"644a5fe7-0455-4a0e-a485-d4063388b732","Type":"ContainerDied","Data":"35fa1d0e7d7fbe04eb5047a1d4018724f2cda274db0e08aa19e68eb965dcba26"}
Oct 30 00:45:30.127699 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:30.127479    6099 scope.go:122] "RemoveContainer" containerID="724d19ae3c3678d60d2447f0a1924d7a618ddd22af5af36cc6f8824f1e67b649"
Oct 30 00:45:30.137118 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:30.137101    6099 scope.go:122] "RemoveContainer" containerID="724d19ae3c3678d60d2447f0a1924d7a618ddd22af5af36cc6f8824f1e67b649"
Oct 30 00:45:30.137402 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:45:30.137384    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724d19ae3c3678d60d2447f0a1924d7a618ddd22af5af36cc6f8824f1e67b649\": container with ID starting with 724d19ae3c3678d60d2447f0a1924d7a618ddd22af5af36cc6f8824f1e67b649 not found: ID does not exist" containerID="724d19ae3c3678d60d2447f0a1924d7a618ddd22af5af36cc6f8824f1e67b649"
Oct 30 00:45:30.137484 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:30.137408    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724d19ae3c3678d60d2447f0a1924d7a618ddd22af5af36cc6f8824f1e67b649"} err="failed to get container status \"724d19ae3c3678d60d2447f0a1924d7a618ddd22af5af36cc6f8824f1e67b649\": rpc error: code = NotFound desc = could not find container \"724d19ae3c3678d60d2447f0a1924d7a618ddd22af5af36cc6f8824f1e67b649\": container with ID starting with 724d19ae3c3678d60d2447f0a1924d7a618ddd22af5af36cc6f8824f1e67b649 not found: ID does not exist"
Oct 30 00:45:30.239144 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:30.239097    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-d8c8ccb-5r5q2"]
Oct 30 00:45:30.242352 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:30.242329    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-d8c8ccb-5r5q2"]
Oct 30 00:45:30.260463 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:45:30.260433    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="644a5fe7-0455-4a0e-a485-d4063388b732" path="/var/lib/kubelet/pods/644a5fe7-0455-4a0e-a485-d4063388b732/volumes"
Oct 30 00:45:46.628582 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:45:46.628541    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:46:02.498052 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.498019    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"]
Oct 30 00:46:02.498721 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.498689    6099 cpu_manager.go:414] "containerMap: removing container" podUID="aac79ba0-cc4e-4848-8e8d-97722de763a3" containerName="cleanup"
Oct 30 00:46:02.498835 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.498723    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac79ba0-cc4e-4848-8e8d-97722de763a3" containerName="cleanup"
Oct 30 00:46:02.498835 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.498743    6099 cpu_manager.go:414] "containerMap: removing container" podUID="644a5fe7-0455-4a0e-a485-d4063388b732" containerName="manager"
Oct 30 00:46:02.498835 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.498750    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="644a5fe7-0455-4a0e-a485-d4063388b732" containerName="manager"
Oct 30 00:46:02.499024 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.498896    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="644a5fe7-0455-4a0e-a485-d4063388b732" containerName="manager"
Oct 30 00:46:02.502696 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.502673    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.518157 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.518139    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-40ac41-openshift-default-dockercfg-kr4bx\""
Oct 30 00:46:02.529831 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.529810    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"]
Oct 30 00:46:02.590657 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.590619    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.590827 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.590663    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.590827 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.590770    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a38635-cb84-404c-9502-de7b43239b8e-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.590916 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.590893    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k8bl\" (UniqueName: \"kubernetes.io/projected/80a38635-cb84-404c-9502-de7b43239b8e-kube-api-access-5k8bl\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.590971 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.590930    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.591016 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.590990    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a38635-cb84-404c-9502-de7b43239b8e-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.591108 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.591092    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a38635-cb84-404c-9502-de7b43239b8e-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.591181 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.591163    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.591227 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.591212    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.631131 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.631098    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"]
Oct 30 00:46:02.635171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.635149    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.654619 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.654597    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5062f6-openshift-default-dockercfg-9bf6d\""
Oct 30 00:46:02.665325 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.665303    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"]
Oct 30 00:46:02.692240 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.692215    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.692376 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.692251    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.692376 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.692268    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a38635-cb84-404c-9502-de7b43239b8e-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.692376 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.692288    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.692584 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.692571    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.692729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.692687    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.692810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.692733    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a38635-cb84-404c-9502-de7b43239b8e-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.692810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.692798    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.692916 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.692834    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.692916 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.692868    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.693021 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.692936    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.693021 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.692978    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.693135 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.693019    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a38635-cb84-404c-9502-de7b43239b8e-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.693135 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.693054    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.693135 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.693093    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9pp2\" (UniqueName: \"kubernetes.io/projected/19331f3a-39f8-46af-8b24-ccfa677abecf-kube-api-access-g9pp2\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.693285 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.693136    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.693285 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.693187    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.693285 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.693197    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.693285 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.693255    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19331f3a-39f8-46af-8b24-ccfa677abecf-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.693285 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.693270    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.693529 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.693272    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a38635-cb84-404c-9502-de7b43239b8e-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.693529 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.693291    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.693529 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.693362    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8bl\" (UniqueName: \"kubernetes.io/projected/80a38635-cb84-404c-9502-de7b43239b8e-kube-api-access-5k8bl\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.696063 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.696036    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.697099 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.696527    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a38635-cb84-404c-9502-de7b43239b8e-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.713817 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.713794    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a38635-cb84-404c-9502-de7b43239b8e-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.713958 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.713941    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k8bl\" (UniqueName: \"kubernetes.io/projected/80a38635-cb84-404c-9502-de7b43239b8e-kube-api-access-5k8bl\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.791394 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.791368    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"]
Oct 30 00:46:02.794262 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.794236    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.794422 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.794271    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.794422 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.794324    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.794422 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.794394    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.794586 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.794432    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9pp2\" (UniqueName: \"kubernetes.io/projected/19331f3a-39f8-46af-8b24-ccfa677abecf-kube-api-access-g9pp2\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.794586 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.794469    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.794586 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.794518    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.794586 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.794546    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19331f3a-39f8-46af-8b24-ccfa677abecf-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.794818 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.794606    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.794870 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.794832    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.794978 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.794929    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.795038 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.794977    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.795094 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.795050    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.795975 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.795946    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19331f3a-39f8-46af-8b24-ccfa677abecf-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.796995 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.796969    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.797093 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.796968    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.797917 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.797900    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.816437 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.816415    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:02.818972 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.818948    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-beb8d5-openshift-default-dockercfg-kjcx7\""
Oct 30 00:46:02.819162 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.819140    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9pp2\" (UniqueName: \"kubernetes.io/projected/19331f3a-39f8-46af-8b24-ccfa677abecf-kube-api-access-g9pp2\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.819958 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.819938    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.825665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.825641    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"]
Oct 30 00:46:02.895532 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.895496    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.895689 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.895575    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv2jh\" (UniqueName: \"kubernetes.io/projected/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-kube-api-access-bv2jh\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.895781 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.895688    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.895844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.895789    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.895909 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.895843    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.895909 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.895885    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.895909 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.895906    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.896076 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.895979    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.896076 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.896038    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.950528 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.950494    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:02.956022 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.955442    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"]
Oct 30 00:46:02.959917 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.959895    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:02.989216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.989183    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"]
Oct 30 00:46:02.997493 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.997457    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.997493 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.997493    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.997733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.997513    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.997733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.997539    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:02.997733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.997577    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.997733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.997601    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:02.997733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.997629    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:02.997733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.997645    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:02.997733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.997660    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.997733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.997681    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:02.998180 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.997748    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2jh\" (UniqueName: \"kubernetes.io/projected/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-kube-api-access-bv2jh\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.998180 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.997778    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:02.998180 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.997795    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:02.998180 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.997812    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:02.998180 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.997830    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.998180 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.997856    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr5sk\" (UniqueName: \"kubernetes.io/projected/24e7a71a-4aa1-46d0-a903-a480cac72a2b-kube-api-access-vr5sk\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:02.998180 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.997878    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.998180 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.997900    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.998601 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.998415    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.998601 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.998590    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.999172 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.999087    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:02.999172 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:02.999112    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:03.000119 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.000094    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:03.002841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.002814    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:03.003412 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.003389    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:03.014033 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.014007    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6add35-openshift-default-dockercfg-5zrdb\""
Oct 30 00:46:03.023763 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.023732    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv2jh\" (UniqueName: \"kubernetes.io/projected/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-kube-api-access-bv2jh\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:03.024262 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.024244    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:03.098954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.098565    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.098954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.098625    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.098954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.098653    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.098954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.098687    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.098954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.098778    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.098954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.098806    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.098954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.098837    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.098954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.098880    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr5sk\" (UniqueName: \"kubernetes.io/projected/24e7a71a-4aa1-46d0-a903-a480cac72a2b-kube-api-access-vr5sk\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.098954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.098959    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.100132 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.099154    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.100132 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.099364    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.100132 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.099626    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.100132 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.099873    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.100490 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.100442    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.100762 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.100726    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"]
Oct 30 00:46:03.102347 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.102303    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.102927 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.102905    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.106236 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.106217    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.120356 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.120340    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b3593a-openshift-default-dockercfg-knq67\""
Oct 30 00:46:03.125154 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.125134    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:03.125610 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.125587    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr5sk\" (UniqueName: \"kubernetes.io/projected/24e7a71a-4aa1-46d0-a903-a480cac72a2b-kube-api-access-vr5sk\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.125768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.125745    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.129686 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.129660    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"]
Oct 30 00:46:03.136086 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.136063    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"]
Oct 30 00:46:03.142022 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:46:03.141996    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a38635_cb84_404c_9502_de7b43239b8e.slice/crio-524d3028bef5ce65bcf3e7bc880b8f6d6522892b804b0e0b914349ce438b63b7 WatchSource:0}: Error finding container 524d3028bef5ce65bcf3e7bc880b8f6d6522892b804b0e0b914349ce438b63b7: Status 404 returned error can't find the container with id 524d3028bef5ce65bcf3e7bc880b8f6d6522892b804b0e0b914349ce438b63b7
Oct 30 00:46:03.144733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.144688    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:03.144807 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.144768    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:03.144878 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.144805    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:03.201031 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.200996    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.201164 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.201083    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.201164 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.201152    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c02eae9-c5d9-459d-8969-b6d3686999d3-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.201277 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.201192    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.201277 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.201239    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scwhw\" (UniqueName: \"kubernetes.io/projected/1c02eae9-c5d9-459d-8969-b6d3686999d3-kube-api-access-scwhw\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.201277 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.201272    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.201441 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.201347    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.201441 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.201381    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.201441 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.201405    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.257237 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.257205    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"]
Oct 30 00:46:03.260954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.260930    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.281139 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.281099    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-89a676-openshift-default-dockercfg-2l6sv\""
Oct 30 00:46:03.282583 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.282555    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"]
Oct 30 00:46:03.286178 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.285959    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:03.286178 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:46:03.285986    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19331f3a_39f8_46af_8b24_ccfa677abecf.slice/crio-dc669d150df2112318239373b6097ed1fd69235f9f7b2c4f382475eb3f1c98fa WatchSource:0}: Error finding container dc669d150df2112318239373b6097ed1fd69235f9f7b2c4f382475eb3f1c98fa: Status 404 returned error can't find the container with id dc669d150df2112318239373b6097ed1fd69235f9f7b2c4f382475eb3f1c98fa
Oct 30 00:46:03.288497 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.288394    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:03.288595 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.288552    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:03.288668 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.288598    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:03.298517 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.296403    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"]
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.302415    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.302475    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scwhw\" (UniqueName: \"kubernetes.io/projected/1c02eae9-c5d9-459d-8969-b6d3686999d3-kube-api-access-scwhw\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.302515    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.302541    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.302593    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.302613    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.302638    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.302653    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.302674    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.302739    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.302758    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.302774    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc360f77-72ad-4729-a4e7-bf542d69b142-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.302800    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gznp\" (UniqueName: \"kubernetes.io/projected/cc360f77-72ad-4729-a4e7-bf542d69b142-kube-api-access-4gznp\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.302824    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.302855    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.302874    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.302919    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.302945    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c02eae9-c5d9-459d-8969-b6d3686999d3-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.303258    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.303548    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c02eae9-c5d9-459d-8969-b6d3686999d3-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.304778    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.305337    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.306253    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.306518    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.308145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.307267    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.319277 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.317563    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p" event={"ID":"19331f3a-39f8-46af-8b24-ccfa677abecf","Type":"ContainerStarted","Data":"dc669d150df2112318239373b6097ed1fd69235f9f7b2c4f382475eb3f1c98fa"}
Oct 30 00:46:03.320767 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.320469    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr" event={"ID":"80a38635-cb84-404c-9502-de7b43239b8e","Type":"ContainerStarted","Data":"f00b518c3e4126a78dbd468479f84e385d5a4656ca492dda7dc85397350e3342"}
Oct 30 00:46:03.320767 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.320528    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr" event={"ID":"80a38635-cb84-404c-9502-de7b43239b8e","Type":"ContainerStarted","Data":"524d3028bef5ce65bcf3e7bc880b8f6d6522892b804b0e0b914349ce438b63b7"}
Oct 30 00:46:03.330879 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.330493    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scwhw\" (UniqueName: \"kubernetes.io/projected/1c02eae9-c5d9-459d-8969-b6d3686999d3-kube-api-access-scwhw\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.333734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.333660    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.367183 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.365645    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr" podStartSLOduration=1.3656281350000001 podStartE2EDuration="1.365628135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:03.365557614 +0000 UTC m=+657.222465373" watchObservedRunningTime="2026-10-30 00:46:03.365628135 +0000 UTC m=+657.222535894"
Oct 30 00:46:03.403688 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.403654    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.403837 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.403733    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.403837 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.403806    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.403961 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.403874    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.403961 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.403898    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.404066 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.403964    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.404066 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.403995    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.404066 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.404011    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc360f77-72ad-4729-a4e7-bf542d69b142-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.404066 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.404041    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gznp\" (UniqueName: \"kubernetes.io/projected/cc360f77-72ad-4729-a4e7-bf542d69b142-kube-api-access-4gznp\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.404849 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.404823    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.405217 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.405191    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.408056 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.407174    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.408056 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.407949    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.409094 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.409050    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc360f77-72ad-4729-a4e7-bf542d69b142-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.409832 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.409812    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.410543 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.410497    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.420814 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.420602    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:03.449665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.449544    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"]
Oct 30 00:46:03.458150 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.453183    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gznp\" (UniqueName: \"kubernetes.io/projected/cc360f77-72ad-4729-a4e7-bf542d69b142-kube-api-access-4gznp\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.458150 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.453805    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.459177 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:46:03.459133    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de29f4b_4f9e_4e67_a43a_f414dc775ee3.slice/crio-ff516610eb85a249e75efdfeca4659b76c25de44de589dd98a615ad84d667dd6 WatchSource:0}: Error finding container ff516610eb85a249e75efdfeca4659b76c25de44de589dd98a615ad84d667dd6: Status 404 returned error can't find the container with id ff516610eb85a249e75efdfeca4659b76c25de44de589dd98a615ad84d667dd6
Oct 30 00:46:03.461368 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.461332    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:03.461454 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.461414    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:03.461512 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.461455    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:03.579509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.579475    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:03.618052 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.617651    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"]
Oct 30 00:46:03.642368 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:46:03.641984    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e7a71a_4aa1_46d0_a903_a480cac72a2b.slice/crio-7a30f2210cdb48250fb7fd4b89ccc8639f3ddce42851965a0aa8a0bddbb7bb2b WatchSource:0}: Error finding container 7a30f2210cdb48250fb7fd4b89ccc8639f3ddce42851965a0aa8a0bddbb7bb2b: Status 404 returned error can't find the container with id 7a30f2210cdb48250fb7fd4b89ccc8639f3ddce42851965a0aa8a0bddbb7bb2b
Oct 30 00:46:03.672336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.672287    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:03.679008 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.678965    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:03.679266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.679241    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:03.772638 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.772612    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"]
Oct 30 00:46:03.781516 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.780330    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:03.781516 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.780415    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:03.781516 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.780459    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:03.816919 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.816895    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:03.939865 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.939332    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"]
Oct 30 00:46:03.943277 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:46:03.943222    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc360f77_72ad_4729_a4e7_bf542d69b142.slice/crio-2dbdec5685cbd0c6cbbd85b10ee336bba72df5a2f71670c800abe98d94754c50 WatchSource:0}: Error finding container 2dbdec5685cbd0c6cbbd85b10ee336bba72df5a2f71670c800abe98d94754c50: Status 404 returned error can't find the container with id 2dbdec5685cbd0c6cbbd85b10ee336bba72df5a2f71670c800abe98d94754c50
Oct 30 00:46:03.946247 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.946207    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:03.946337 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.946292    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:03.946446 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:03.946335    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:04.327271 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.327235    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p" event={"ID":"19331f3a-39f8-46af-8b24-ccfa677abecf","Type":"ContainerStarted","Data":"151d441a1fa03d80f442bf6b6a3e8efb2b26e99b9fa183840e413776eb699860"}
Oct 30 00:46:04.328838 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.328810    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw" event={"ID":"1c02eae9-c5d9-459d-8969-b6d3686999d3","Type":"ContainerStarted","Data":"afe80ec41b8c299ca906e561621eaf27a6e296ddbef21a23e8d64c5ae6c796cc"}
Oct 30 00:46:04.328983 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.328855    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw" event={"ID":"1c02eae9-c5d9-459d-8969-b6d3686999d3","Type":"ContainerStarted","Data":"8975a69701e9b09228699c713542e200d37734fd991dc40637525c0c30eaa2ce"}
Oct 30 00:46:04.330260 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.330233    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh" event={"ID":"6de29f4b-4f9e-4e67-a43a-f414dc775ee3","Type":"ContainerStarted","Data":"4747ec3dae7bc350f37088743008ae26ea8c4f5860526badd4f50b04ae133c0b"}
Oct 30 00:46:04.330398 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.330278    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh" event={"ID":"6de29f4b-4f9e-4e67-a43a-f414dc775ee3","Type":"ContainerStarted","Data":"ff516610eb85a249e75efdfeca4659b76c25de44de589dd98a615ad84d667dd6"}
Oct 30 00:46:04.331775 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.331745    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w" event={"ID":"cc360f77-72ad-4729-a4e7-bf542d69b142","Type":"ContainerStarted","Data":"9195d5f4afb0146b39bad6c325ade3192ec9f7933ed4cefb337912cbeb1d44e8"}
Oct 30 00:46:04.331880 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.331795    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w" event={"ID":"cc360f77-72ad-4729-a4e7-bf542d69b142","Type":"ContainerStarted","Data":"2dbdec5685cbd0c6cbbd85b10ee336bba72df5a2f71670c800abe98d94754c50"}
Oct 30 00:46:04.333187 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.333163    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp" event={"ID":"24e7a71a-4aa1-46d0-a903-a480cac72a2b","Type":"ContainerStarted","Data":"bbacec21207034f88fa78397321b6ad51b8379ebc8ae7cd5c5698852bec3ab64"}
Oct 30 00:46:04.333272 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.333209    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp" event={"ID":"24e7a71a-4aa1-46d0-a903-a480cac72a2b","Type":"ContainerStarted","Data":"7a30f2210cdb48250fb7fd4b89ccc8639f3ddce42851965a0aa8a0bddbb7bb2b"}
Oct 30 00:46:04.367982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.367931    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p" podStartSLOduration=2.367917331 podStartE2EDuration="2.367917331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:04.366946376 +0000 UTC m=+658.223854143" watchObservedRunningTime="2026-10-30 00:46:04.367917331 +0000 UTC m=+658.224825153"
Oct 30 00:46:04.410130 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.410079    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w" podStartSLOduration=1.4100620240000001 podStartE2EDuration="1.410062024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:04.408731496 +0000 UTC m=+658.265639247" watchObservedRunningTime="2026-10-30 00:46:04.410062024 +0000 UTC m=+658.266969779"
Oct 30 00:46:04.420981 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.420954    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:04.425632 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.425609    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:04.454767 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.454723    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp" podStartSLOduration=2.454694167 podStartE2EDuration="2.454694167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:04.453108356 +0000 UTC m=+658.310016110" watchObservedRunningTime="2026-10-30 00:46:04.454694167 +0000 UTC m=+658.311601922"
Oct 30 00:46:04.493718 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.493666    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh" podStartSLOduration=2.493650686 podStartE2EDuration="2.493650686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:04.49256855 +0000 UTC m=+658.349476306" watchObservedRunningTime="2026-10-30 00:46:04.493650686 +0000 UTC m=+658.350558458"
Oct 30 00:46:04.536506 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.536452    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw" podStartSLOduration=1.536437395 podStartE2EDuration="1.536437395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:04.534929833 +0000 UTC m=+658.391837588" watchObservedRunningTime="2026-10-30 00:46:04.536437395 +0000 UTC m=+658.393345149"
Oct 30 00:46:04.580370 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.580302    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:04.822535 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.822504    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:04.951727 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.951622    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:04.957588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:04.957561    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:05.125605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:05.125574    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:05.131637 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:05.131610    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:05.287243 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:05.287154    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:05.292244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:05.292214    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:05.339882 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:05.339848    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:05.340040 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:05.339896    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:05.340040 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:05.339914    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:05.340040 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:05.339931    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:05.340040 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:05.339947    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:05.340040 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:05.339999    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:05.340040 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:05.340029    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:05.340256 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:05.340057    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:05.340256 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:05.340096    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:05.340256 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:05.340247    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:05.585223 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:05.585186    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:06.342895 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:06.342860    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:06.343291 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:06.343272    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:15.357729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:15.355990    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:46:16.305571 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:16.305531    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h"]
Oct 30 00:46:16.311921 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:16.311891    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h"
Oct 30 00:46:16.328626 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:16.328595    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h"]
Oct 30 00:46:16.338795 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:16.338753    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-89a676\""
Oct 30 00:46:16.338904 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:16.338823    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-89a676-dockercfg-7mcr5\""
Oct 30 00:46:16.421977 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:16.421807    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl5dq\" (UniqueName: \"kubernetes.io/projected/97c0665f-6367-4539-a67d-3cc6b53fec21-kube-api-access-bl5dq\") pod \"payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h\" (UID: \"97c0665f-6367-4539-a67d-3cc6b53fec21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h"
Oct 30 00:46:16.422868 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:16.422838    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c0665f-6367-4539-a67d-3cc6b53fec21-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h\" (UID: \"97c0665f-6367-4539-a67d-3cc6b53fec21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h"
Oct 30 00:46:16.525056 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:16.525013    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl5dq\" (UniqueName: \"kubernetes.io/projected/97c0665f-6367-4539-a67d-3cc6b53fec21-kube-api-access-bl5dq\") pod \"payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h\" (UID: \"97c0665f-6367-4539-a67d-3cc6b53fec21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h"
Oct 30 00:46:16.525279 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:16.525113    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c0665f-6367-4539-a67d-3cc6b53fec21-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h\" (UID: \"97c0665f-6367-4539-a67d-3cc6b53fec21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h"
Oct 30 00:46:16.526427 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:16.526398    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c0665f-6367-4539-a67d-3cc6b53fec21-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h\" (UID: \"97c0665f-6367-4539-a67d-3cc6b53fec21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h"
Oct 30 00:46:16.546131 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:16.546105    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl5dq\" (UniqueName: \"kubernetes.io/projected/97c0665f-6367-4539-a67d-3cc6b53fec21-kube-api-access-bl5dq\") pod \"payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h\" (UID: \"97c0665f-6367-4539-a67d-3cc6b53fec21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h"
Oct 30 00:46:16.628868 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:16.628788    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h"
Oct 30 00:46:16.630407 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:46:16.630376    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:46:16.949771 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:16.949673    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h"]
Oct 30 00:46:16.954252 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:46:16.954223    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c0665f_6367_4539_a67d_3cc6b53fec21.slice/crio-75a1e8bc46511cbcdb7facbd59881bb90fdc6bb779f5af5f1f1eb336bc2bc964 WatchSource:0}: Error finding container 75a1e8bc46511cbcdb7facbd59881bb90fdc6bb779f5af5f1f1eb336bc2bc964: Status 404 returned error can't find the container with id 75a1e8bc46511cbcdb7facbd59881bb90fdc6bb779f5af5f1f1eb336bc2bc964
Oct 30 00:46:17.188731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.186739    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr"]
Oct 30 00:46:17.195375 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.195343    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr"
Oct 30 00:46:17.202909 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.202881    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr"]
Oct 30 00:46:17.207834 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.207807    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-2b4105\""
Oct 30 00:46:17.211038 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.211014    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-2b4105-dockercfg-rdx4f\""
Oct 30 00:46:17.310809 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.310780    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt"]
Oct 30 00:46:17.315123 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.315104    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt"]
Oct 30 00:46:17.315289 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.315216    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt"
Oct 30 00:46:17.351845 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.351817    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354c452d-1616-4d33-9d18-a44f6fc76366-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr\" (UID: \"354c452d-1616-4d33-9d18-a44f6fc76366\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr"
Oct 30 00:46:17.352004 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.351947    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h479l\" (UniqueName: \"kubernetes.io/projected/61780fe7-50f0-4f30-989c-03e165d46adf-kube-api-access-h479l\") pod \"payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt\" (UID: \"61780fe7-50f0-4f30-989c-03e165d46adf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt"
Oct 30 00:46:17.352048 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.352020    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n6mf\" (UniqueName: \"kubernetes.io/projected/354c452d-1616-4d33-9d18-a44f6fc76366-kube-api-access-5n6mf\") pod \"payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr\" (UID: \"354c452d-1616-4d33-9d18-a44f6fc76366\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr"
Oct 30 00:46:17.352122 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.352104    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61780fe7-50f0-4f30-989c-03e165d46adf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt\" (UID: \"61780fe7-50f0-4f30-989c-03e165d46adf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt"
Oct 30 00:46:17.427910 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.427871    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h" event={"ID":"97c0665f-6367-4539-a67d-3cc6b53fec21","Type":"ContainerStarted","Data":"081da4fcde248d7be0ca0b482aa8832947123a9d3bf2d794b11f0fa3275a1437"}
Oct 30 00:46:17.428438 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.427936    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h" event={"ID":"97c0665f-6367-4539-a67d-3cc6b53fec21","Type":"ContainerStarted","Data":"75a1e8bc46511cbcdb7facbd59881bb90fdc6bb779f5af5f1f1eb336bc2bc964"}
Oct 30 00:46:17.428438 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.428011    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h"
Oct 30 00:46:17.452790 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.452692    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354c452d-1616-4d33-9d18-a44f6fc76366-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr\" (UID: \"354c452d-1616-4d33-9d18-a44f6fc76366\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr"
Oct 30 00:46:17.452790 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.452774    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h479l\" (UniqueName: \"kubernetes.io/projected/61780fe7-50f0-4f30-989c-03e165d46adf-kube-api-access-h479l\") pod \"payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt\" (UID: \"61780fe7-50f0-4f30-989c-03e165d46adf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt"
Oct 30 00:46:17.453000 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.452812    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n6mf\" (UniqueName: \"kubernetes.io/projected/354c452d-1616-4d33-9d18-a44f6fc76366-kube-api-access-5n6mf\") pod \"payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr\" (UID: \"354c452d-1616-4d33-9d18-a44f6fc76366\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr"
Oct 30 00:46:17.453000 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.452873    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61780fe7-50f0-4f30-989c-03e165d46adf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt\" (UID: \"61780fe7-50f0-4f30-989c-03e165d46adf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt"
Oct 30 00:46:17.453467 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.453440    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354c452d-1616-4d33-9d18-a44f6fc76366-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr\" (UID: \"354c452d-1616-4d33-9d18-a44f6fc76366\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr"
Oct 30 00:46:17.453590 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.453572    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61780fe7-50f0-4f30-989c-03e165d46adf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt\" (UID: \"61780fe7-50f0-4f30-989c-03e165d46adf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt"
Oct 30 00:46:17.485729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.482247    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h" podStartSLOduration=1.482228978 podStartE2EDuration="1.482228978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:17.481328822 +0000 UTC m=+671.338236578" watchObservedRunningTime="2026-10-30 00:46:17.482228978 +0000 UTC m=+671.339136735"
Oct 30 00:46:17.491600 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.491572    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h479l\" (UniqueName: \"kubernetes.io/projected/61780fe7-50f0-4f30-989c-03e165d46adf-kube-api-access-h479l\") pod \"payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt\" (UID: \"61780fe7-50f0-4f30-989c-03e165d46adf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt"
Oct 30 00:46:17.498059 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.498036    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n6mf\" (UniqueName: \"kubernetes.io/projected/354c452d-1616-4d33-9d18-a44f6fc76366-kube-api-access-5n6mf\") pod \"payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr\" (UID: \"354c452d-1616-4d33-9d18-a44f6fc76366\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr"
Oct 30 00:46:17.509909 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.509886    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr"
Oct 30 00:46:17.629054 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.629015    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt"
Oct 30 00:46:17.831897 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.831715    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr"]
Oct 30 00:46:17.834863 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:46:17.834822    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354c452d_1616_4d33_9d18_a44f6fc76366.slice/crio-bebc199a9d92e2840c8cedc4d030b0965d28484ec74ceb4b760babd8c05f73d6 WatchSource:0}: Error finding container bebc199a9d92e2840c8cedc4d030b0965d28484ec74ceb4b760babd8c05f73d6: Status 404 returned error can't find the container with id bebc199a9d92e2840c8cedc4d030b0965d28484ec74ceb4b760babd8c05f73d6
Oct 30 00:46:17.957051 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:17.956939    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt"]
Oct 30 00:46:17.959226 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:46:17.959200    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61780fe7_50f0_4f30_989c_03e165d46adf.slice/crio-63f90e014ca73b2c08b3e3a36edc574fe916d56f5920ef685258ba102fbd933d WatchSource:0}: Error finding container 63f90e014ca73b2c08b3e3a36edc574fe916d56f5920ef685258ba102fbd933d: Status 404 returned error can't find the container with id 63f90e014ca73b2c08b3e3a36edc574fe916d56f5920ef685258ba102fbd933d
Oct 30 00:46:18.433900 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:18.433862    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr" event={"ID":"354c452d-1616-4d33-9d18-a44f6fc76366","Type":"ContainerStarted","Data":"030ee7d7bc1dfd3d6150d889b785b249161e12d8010c6903639a51550b636745"}
Oct 30 00:46:18.434330 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:18.433922    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr" event={"ID":"354c452d-1616-4d33-9d18-a44f6fc76366","Type":"ContainerStarted","Data":"bebc199a9d92e2840c8cedc4d030b0965d28484ec74ceb4b760babd8c05f73d6"}
Oct 30 00:46:18.434330 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:18.433952    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr"
Oct 30 00:46:18.435606 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:18.435568    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt" event={"ID":"61780fe7-50f0-4f30-989c-03e165d46adf","Type":"ContainerStarted","Data":"e813d54d5bf107aa6a67f4c8b4179365ea39b8456237216e49c85027e710b8bb"}
Oct 30 00:46:18.435781 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:18.435628    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt"
Oct 30 00:46:18.435781 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:18.435655    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt" event={"ID":"61780fe7-50f0-4f30-989c-03e165d46adf","Type":"ContainerStarted","Data":"63f90e014ca73b2c08b3e3a36edc574fe916d56f5920ef685258ba102fbd933d"}
Oct 30 00:46:18.487729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:18.487573    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr" podStartSLOduration=1.487555618 podStartE2EDuration="1.487555618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:18.469036297 +0000 UTC m=+672.325944055" watchObservedRunningTime="2026-10-30 00:46:18.487555618 +0000 UTC m=+672.344463374"
Oct 30 00:46:18.515152 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:18.515102    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt" podStartSLOduration=1.5150879860000002 podStartE2EDuration="1.515087986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:18.513671014 +0000 UTC m=+672.370578771" watchObservedRunningTime="2026-10-30 00:46:18.515087986 +0000 UTC m=+672.371995740"
Oct 30 00:46:24.435728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:24.433793    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:46:28.439463 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:28.439424    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h"
Oct 30 00:46:29.444643 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:29.443970    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt"
Oct 30 00:46:29.445519 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:29.445493    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr"
Oct 30 00:46:30.659472 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:30.657745    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:46:31.872382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:31.871108    6099 generic.go:356] "Generic (PLEG): container finished" podID="80a38635-cb84-404c-9502-de7b43239b8e" containerID="f00b518c3e4126a78dbd468479f84e385d5a4656ca492dda7dc85397350e3342" exitCode=0
Oct 30 00:46:31.872382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:31.871199    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr" event={"ID":"80a38635-cb84-404c-9502-de7b43239b8e","Type":"ContainerDied","Data":"f00b518c3e4126a78dbd468479f84e385d5a4656ca492dda7dc85397350e3342"}
Oct 30 00:46:31.872382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:31.871644    6099 scope.go:122] "RemoveContainer" containerID="f00b518c3e4126a78dbd468479f84e385d5a4656ca492dda7dc85397350e3342"
Oct 30 00:46:31.875065 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:31.874920    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:31.877729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:31.875981    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:31.877729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:31.876305    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:32.918729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:32.918621    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr" event={"ID":"80a38635-cb84-404c-9502-de7b43239b8e","Type":"ContainerStarted","Data":"71f282eb405fe0c8f715a5f2847c21d04a0bdad3bda5c881fbc614823dbc92cf"}
Oct 30 00:46:33.820729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:33.818898    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:33.820729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:33.820274    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 00:46:33.820729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:33.820328    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr" podUID="80a38635-cb84-404c-9502-de7b43239b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 00:46:34.820747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:34.820057    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 00:46:34.820747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:34.820108    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr" podUID="80a38635-cb84-404c-9502-de7b43239b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 00:46:35.339233 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:35.339191    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:36.109336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:35.451516    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": read tcp 10.134.0.2:54966->10.134.0.46:15021: read: connection reset by peer" start-of-body=
Oct 30 00:46:36.109336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:35.451572    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp" podUID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": read tcp 10.134.0.2:54966->10.134.0.46:15021: read: connection reset by peer"
Oct 30 00:46:36.109336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:35.818147    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 00:46:36.109336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:35.818206    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr" podUID="80a38635-cb84-404c-9502-de7b43239b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 00:46:36.109336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:35.941285    6099 generic.go:356] "Generic (PLEG): container finished" podID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" containerID="bbacec21207034f88fa78397321b6ad51b8379ebc8ae7cd5c5698852bec3ab64" exitCode=0
Oct 30 00:46:36.109336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:35.941409    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp" event={"ID":"24e7a71a-4aa1-46d0-a903-a480cac72a2b","Type":"ContainerDied","Data":"bbacec21207034f88fa78397321b6ad51b8379ebc8ae7cd5c5698852bec3ab64"}
Oct 30 00:46:36.109336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:35.941815    6099 scope.go:122] "RemoveContainer" containerID="bbacec21207034f88fa78397321b6ad51b8379ebc8ae7cd5c5698852bec3ab64"
Oct 30 00:46:36.109336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:35.943674    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:36.109336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:35.943757    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:36.109336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:35.943810    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:36.135560 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:36.135524    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": read tcp 10.134.0.2:44684->10.134.0.45:15021: read: connection reset by peer" start-of-body=
Oct 30 00:46:36.135694 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:36.135595    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh" podUID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": read tcp 10.134.0.2:44684->10.134.0.45:15021: read: connection reset by peer"
Oct 30 00:46:36.383185 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:36.379879    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:46:36.383185 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:36.379951    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw" podUID="1c02eae9-c5d9-459d-8969-b6d3686999d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:46:36.386819 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:36.384796    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:46:36.386819 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:36.384850    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p" podUID="19331f3a-39f8-46af-8b24-ccfa677abecf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:46:36.766785 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:36.761323    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:46:36.824731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:36.824063    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 00:46:36.824731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:36.824114    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr" podUID="80a38635-cb84-404c-9502-de7b43239b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 00:46:36.956745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:36.955488    6099 generic.go:356] "Generic (PLEG): container finished" podID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerID="4747ec3dae7bc350f37088743008ae26ea8c4f5860526badd4f50b04ae133c0b" exitCode=0
Oct 30 00:46:36.956745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:36.955600    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh" event={"ID":"6de29f4b-4f9e-4e67-a43a-f414dc775ee3","Type":"ContainerDied","Data":"4747ec3dae7bc350f37088743008ae26ea8c4f5860526badd4f50b04ae133c0b"}
Oct 30 00:46:36.956745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:36.956126    6099 scope.go:122] "RemoveContainer" containerID="4747ec3dae7bc350f37088743008ae26ea8c4f5860526badd4f50b04ae133c0b"
Oct 30 00:46:36.958464 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:36.958423    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:36.958591 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:36.958521    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:36.958591 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:36.958568    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:36.959863 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:36.959837    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp" event={"ID":"24e7a71a-4aa1-46d0-a903-a480cac72a2b","Type":"ContainerStarted","Data":"9a861b5c79b98757e8b1994dd7cde16f934741c92027facdec5bd733794b6b98"}
Oct 30 00:46:37.300740 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:37.291741    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:37.309733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:37.301389    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 00:46:37.309733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:37.301446    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp" podUID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 00:46:37.819545 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:37.819105    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 00:46:37.819545 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:37.819168    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr" podUID="80a38635-cb84-404c-9502-de7b43239b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 00:46:38.008931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:38.008091    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh" event={"ID":"6de29f4b-4f9e-4e67-a43a-f414dc775ee3","Type":"ContainerStarted","Data":"ea02c4366bf188a3dd41c2f3ca0bbabc289fe40ef39602a7f995477e11fd72c1"}
Oct 30 00:46:38.139743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:38.129954    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:38.139743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:38.130072    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:46:38.139743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:38.130113    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh" podUID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 00:46:38.291817 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:38.291782    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 00:46:38.292089 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:38.292063    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp" podUID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 00:46:39.134729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:39.134019    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:46:39.134729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:39.134075    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh" podUID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 00:46:39.291731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:39.291115    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 00:46:39.291731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:39.291173    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp" podUID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 00:46:39.428659 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:39.428546    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:39.431828 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:39.431797    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 00:46:40.126746 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:40.126681    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:46:40.126955 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:40.126777    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh" podUID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 00:46:40.302026 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:40.301995    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 00:46:40.302605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:40.302562    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp" podUID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 00:46:41.041868 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:41.040815    6099 generic.go:356] "Generic (PLEG): container finished" podID="1c02eae9-c5d9-459d-8969-b6d3686999d3" containerID="afe80ec41b8c299ca906e561621eaf27a6e296ddbef21a23e8d64c5ae6c796cc" exitCode=0
Oct 30 00:46:41.041868 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:41.040920    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw" event={"ID":"1c02eae9-c5d9-459d-8969-b6d3686999d3","Type":"ContainerDied","Data":"afe80ec41b8c299ca906e561621eaf27a6e296ddbef21a23e8d64c5ae6c796cc"}
Oct 30 00:46:41.041868 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:41.041284    6099 scope.go:122] "RemoveContainer" containerID="afe80ec41b8c299ca906e561621eaf27a6e296ddbef21a23e8d64c5ae6c796cc"
Oct 30 00:46:41.042816 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:41.042735    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:41.042816 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:41.042813    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:41.042956 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:41.042852    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:41.044073 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:41.044045    6099 generic.go:356] "Generic (PLEG): container finished" podID="19331f3a-39f8-46af-8b24-ccfa677abecf" containerID="151d441a1fa03d80f442bf6b6a3e8efb2b26e99b9fa183840e413776eb699860" exitCode=0
Oct 30 00:46:41.044149 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:41.044101    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p" event={"ID":"19331f3a-39f8-46af-8b24-ccfa677abecf","Type":"ContainerDied","Data":"151d441a1fa03d80f442bf6b6a3e8efb2b26e99b9fa183840e413776eb699860"}
Oct 30 00:46:41.044482 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:41.044465    6099 scope.go:122] "RemoveContainer" containerID="151d441a1fa03d80f442bf6b6a3e8efb2b26e99b9fa183840e413776eb699860"
Oct 30 00:46:41.045992 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:41.045967    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:41.046051 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:41.046037    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:41.046085 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:41.046075    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:41.126136 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:41.126102    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:46:41.126318 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:41.126161    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh" podUID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 00:46:41.290492 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:41.290059    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 00:46:41.290492 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:41.290119    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp" podUID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 00:46:42.094731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:42.094014    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p" event={"ID":"19331f3a-39f8-46af-8b24-ccfa677abecf","Type":"ContainerStarted","Data":"244f27a398e1fa99891444d79abe5b6c0ab86d6a62e429fdadc21e37bebbb848"}
Oct 30 00:46:42.120731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:42.105963    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw" event={"ID":"1c02eae9-c5d9-459d-8969-b6d3686999d3","Type":"ContainerStarted","Data":"371a6aa0a2fa7d8d332cb51ca58472d9965cebf3400f092c0164aa6acc98aea5"}
Oct 30 00:46:42.126101 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:42.126070    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:46:42.126329 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:42.126304    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh" podUID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 00:46:42.293729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:42.293674    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 00:46:42.293936 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:42.293760    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp" podUID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 00:46:42.435292 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:42.435198    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:42.435618 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:42.435596    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 00:46:42.435806 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:42.435780    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw" podUID="1c02eae9-c5d9-459d-8969-b6d3686999d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 00:46:42.952283 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:42.952248    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:42.952699 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:42.952676    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 00:46:42.952894 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:42.952855    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p" podUID="19331f3a-39f8-46af-8b24-ccfa677abecf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 00:46:43.147792 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:43.147758    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:46:43.148388 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:43.148354    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh" podUID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 00:46:43.305729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:43.302079    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:43.305729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:43.303742    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:43.310728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:43.307934    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 00:46:43.428731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:43.428102    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 00:46:43.428731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:43.428162    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw" podUID="1c02eae9-c5d9-459d-8969-b6d3686999d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 00:46:43.956398 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:43.955942    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 00:46:43.956398 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:43.956007    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p" podUID="19331f3a-39f8-46af-8b24-ccfa677abecf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 00:46:44.285632 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:44.284006    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:44.285632 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:44.285423    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:44.290011 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:44.289955    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 00:46:44.427846 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:44.427807    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 00:46:44.428040 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:44.427873    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw" podUID="1c02eae9-c5d9-459d-8969-b6d3686999d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 00:46:44.951276 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:44.951242    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 00:46:44.951467 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:44.951300    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p" podUID="19331f3a-39f8-46af-8b24-ccfa677abecf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 00:46:45.426571 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:45.426142    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 00:46:45.426571 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:45.426202    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw" podUID="1c02eae9-c5d9-459d-8969-b6d3686999d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 00:46:45.951211 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:45.951173    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 00:46:45.951400 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:45.951226    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p" podUID="19331f3a-39f8-46af-8b24-ccfa677abecf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 00:46:46.422092 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.422051    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 00:46:46.422288 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.422103    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw" podUID="1c02eae9-c5d9-459d-8969-b6d3686999d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 00:46:46.510835 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.502936    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"]
Oct 30 00:46:46.515967 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.515941    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.525616 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.525475    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"]
Oct 30 00:46:46.532292 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.532214    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5062f6\"/\"default-dockercfg-cxknj\""
Oct 30 00:46:46.532553 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.532533    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5062f6\"/\"simulator-w6-5062f6-kserve-self-signed-certs\""
Oct 30 00:46:46.538564 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.538467    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5062f6\"/\"openshift-service-ca.crt\""
Oct 30 00:46:46.538564 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.538540    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5062f6\"/\"kube-root-ca.crt\""
Oct 30 00:46:46.539312 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.539275    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-home\") pod \"simulator-w6-5062f6-kserve-55764c85bb-xsv29\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") " pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.539416 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.539319    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-model-cache\") pod \"simulator-w6-5062f6-kserve-55764c85bb-xsv29\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") " pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.539416 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.539374    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b46rt\" (UniqueName: \"kubernetes.io/projected/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-kube-api-access-b46rt\") pod \"simulator-w6-5062f6-kserve-55764c85bb-xsv29\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") " pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.539520 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.539435    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-tls-certs\") pod \"simulator-w6-5062f6-kserve-55764c85bb-xsv29\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") " pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.539520 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.539468    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-dshm\") pod \"simulator-w6-5062f6-kserve-55764c85bb-xsv29\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") " pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.539629 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.539523    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-tmp-dir\") pod \"simulator-w6-5062f6-kserve-55764c85bb-xsv29\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") " pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.641843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.640685    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-model-cache\") pod \"simulator-w6-5062f6-kserve-55764c85bb-xsv29\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") " pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.641843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.640758    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b46rt\" (UniqueName: \"kubernetes.io/projected/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-kube-api-access-b46rt\") pod \"simulator-w6-5062f6-kserve-55764c85bb-xsv29\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") " pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.641843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.640795    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-tls-certs\") pod \"simulator-w6-5062f6-kserve-55764c85bb-xsv29\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") " pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.641843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.640819    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-dshm\") pod \"simulator-w6-5062f6-kserve-55764c85bb-xsv29\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") " pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.641843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.640865    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-tmp-dir\") pod \"simulator-w6-5062f6-kserve-55764c85bb-xsv29\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") " pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.641843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.640969    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-home\") pod \"simulator-w6-5062f6-kserve-55764c85bb-xsv29\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") " pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.641843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.641316    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-home\") pod \"simulator-w6-5062f6-kserve-55764c85bb-xsv29\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") " pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.641843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.641524    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-model-cache\") pod \"simulator-w6-5062f6-kserve-55764c85bb-xsv29\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") " pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.642252 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.642055    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-tmp-dir\") pod \"simulator-w6-5062f6-kserve-55764c85bb-xsv29\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") " pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.644634 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.644606    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-dshm\") pod \"simulator-w6-5062f6-kserve-55764c85bb-xsv29\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") " pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.645642 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.645617    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-tls-certs\") pod \"simulator-w6-5062f6-kserve-55764c85bb-xsv29\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") " pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.646784 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:46:46.646749    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:46:46.677772 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.673216    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b46rt\" (UniqueName: \"kubernetes.io/projected/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-kube-api-access-b46rt\") pod \"simulator-w6-5062f6-kserve-55764c85bb-xsv29\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") " pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.834729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.834439    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:46.952540 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.951253    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 00:46:46.952540 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:46.951310    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p" podUID="19331f3a-39f8-46af-8b24-ccfa677abecf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 00:46:47.217817 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:47.216215    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"]
Oct 30 00:46:47.422667 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:47.422232    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 00:46:47.422667 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:47.422292    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw" podUID="1c02eae9-c5d9-459d-8969-b6d3686999d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 00:46:47.529971 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:47.529346    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:46:47.960436 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:47.960411    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:47.960919 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:47.960894    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:47.961683 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:47.961621    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 00:46:48.208317 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:48.208224    6099 generic.go:356] "Generic (PLEG): container finished" podID="cc360f77-72ad-4729-a4e7-bf542d69b142" containerID="9195d5f4afb0146b39bad6c325ade3192ec9f7933ed4cefb337912cbeb1d44e8" exitCode=0
Oct 30 00:46:48.208636 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:48.208612    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w" event={"ID":"cc360f77-72ad-4729-a4e7-bf542d69b142","Type":"ContainerDied","Data":"9195d5f4afb0146b39bad6c325ade3192ec9f7933ed4cefb337912cbeb1d44e8"}
Oct 30 00:46:48.209279 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:48.209258    6099 scope.go:122] "RemoveContainer" containerID="9195d5f4afb0146b39bad6c325ade3192ec9f7933ed4cefb337912cbeb1d44e8"
Oct 30 00:46:48.211807 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:48.211772    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:48.212015 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:48.211989    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:48.212175 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:48.212152    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:46:48.212815 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:48.211856    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" event={"ID":"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b","Type":"ContainerStarted","Data":"d30d5c9bd2d3de754c957ed0667336034a8bfa55620acd2e9f34ccebae69b3b0"}
Oct 30 00:46:48.213125 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:48.212990    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" event={"ID":"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b","Type":"ContainerStarted","Data":"4561ef4c6e544ac676c1004ba91caeea411658c887ecf9e46448bf98414f0f77"}
Oct 30 00:46:48.293271 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:48.293219    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podStartSLOduration=2.292698285 podStartE2EDuration="2.292698285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:48.290539603 +0000 UTC m=+702.147447359" watchObservedRunningTime="2026-10-30 00:46:48.292698285 +0000 UTC m=+702.149606042"
Oct 30 00:46:48.434403 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:48.432976    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:48.434403 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:48.434268    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:48.452220 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:48.452179    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 00:46:48.843887 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:48.843848    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:49.219345 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:49.219249    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w" event={"ID":"cc360f77-72ad-4729-a4e7-bf542d69b142","Type":"ContainerStarted","Data":"3fd6fcdc43306ca714f166a92425d2fa4f40230e6bf49fe722d7600d0df5f4c4"}
Oct 30 00:46:49.590732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:49.590271    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:49.590732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:49.590444    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:46:49.590732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:49.590493    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w" podUID="cc360f77-72ad-4729-a4e7-bf542d69b142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 00:46:50.582126 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:50.582088    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:46:50.582723 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:50.582671    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w" podUID="cc360f77-72ad-4729-a4e7-bf542d69b142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 00:46:51.343170 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:51.343138    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:51.583730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:51.583142    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:46:51.583730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:51.583210    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w" podUID="cc360f77-72ad-4729-a4e7-bf542d69b142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 00:46:52.586729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:52.582663    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:46:52.586729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:52.582750    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w" podUID="cc360f77-72ad-4729-a4e7-bf542d69b142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 00:46:53.580133 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:53.580096    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:46:53.581244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:53.581203    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w" podUID="cc360f77-72ad-4729-a4e7-bf542d69b142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 00:46:54.589839 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:54.589789    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:54.591794 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:54.591761    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 00:46:56.835589 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:56.835555    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:56.848454 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:56.848426    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:57.274499 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.274420    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:46:57.384124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.384090    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"]
Oct 30 00:46:57.387939 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.387919    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.405883 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.405862    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b3593a\"/\"openshift-service-ca.crt\""
Oct 30 00:46:57.406002 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.405889    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b3593a\"/\"kube-root-ca.crt\""
Oct 30 00:46:57.406101 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.406086    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b3593a\"/\"default-dockercfg-jj8kk\""
Oct 30 00:46:57.406225 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.406211    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b3593a\"/\"simulator-w3-b3593a-kserve-self-signed-certs\""
Oct 30 00:46:57.407190 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.407167    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"]
Oct 30 00:46:57.550748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.550700    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93ef078d-6084-421b-9311-c91a4852be8b-tls-certs\") pod \"simulator-w3-b3593a-kserve-74b95c566f-bb5sk\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") " pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.550901 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.550763    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-home\") pod \"simulator-w3-b3593a-kserve-74b95c566f-bb5sk\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") " pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.550944 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.550891    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-dshm\") pod \"simulator-w3-b3593a-kserve-74b95c566f-bb5sk\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") " pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.550979 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.550947    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-model-cache\") pod \"simulator-w3-b3593a-kserve-74b95c566f-bb5sk\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") " pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.551012 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.550999    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgnfk\" (UniqueName: \"kubernetes.io/projected/93ef078d-6084-421b-9311-c91a4852be8b-kube-api-access-dgnfk\") pod \"simulator-w3-b3593a-kserve-74b95c566f-bb5sk\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") " pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.551059 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.551046    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-tmp-dir\") pod \"simulator-w3-b3593a-kserve-74b95c566f-bb5sk\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") " pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.652363 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.652329    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93ef078d-6084-421b-9311-c91a4852be8b-tls-certs\") pod \"simulator-w3-b3593a-kserve-74b95c566f-bb5sk\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") " pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.652558 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.652378    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-home\") pod \"simulator-w3-b3593a-kserve-74b95c566f-bb5sk\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") " pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.652558 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.652442    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-dshm\") pod \"simulator-w3-b3593a-kserve-74b95c566f-bb5sk\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") " pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.652558 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.652497    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-model-cache\") pod \"simulator-w3-b3593a-kserve-74b95c566f-bb5sk\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") " pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.652558 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.652524    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgnfk\" (UniqueName: \"kubernetes.io/projected/93ef078d-6084-421b-9311-c91a4852be8b-kube-api-access-dgnfk\") pod \"simulator-w3-b3593a-kserve-74b95c566f-bb5sk\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") " pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.652778 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.652572    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-tmp-dir\") pod \"simulator-w3-b3593a-kserve-74b95c566f-bb5sk\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") " pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.652912 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.652881    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-home\") pod \"simulator-w3-b3593a-kserve-74b95c566f-bb5sk\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") " pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.653050 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.652937    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-model-cache\") pod \"simulator-w3-b3593a-kserve-74b95c566f-bb5sk\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") " pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.653050 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.653011    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-tmp-dir\") pod \"simulator-w3-b3593a-kserve-74b95c566f-bb5sk\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") " pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.654719 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.654688    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-dshm\") pod \"simulator-w3-b3593a-kserve-74b95c566f-bb5sk\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") " pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.655084 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.655064    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93ef078d-6084-421b-9311-c91a4852be8b-tls-certs\") pod \"simulator-w3-b3593a-kserve-74b95c566f-bb5sk\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") " pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.673266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.673243    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgnfk\" (UniqueName: \"kubernetes.io/projected/93ef078d-6084-421b-9311-c91a4852be8b-kube-api-access-dgnfk\") pod \"simulator-w3-b3593a-kserve-74b95c566f-bb5sk\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") " pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.700271 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.700251    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:46:57.993606 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:57.993584    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"]
Oct 30 00:46:57.996124 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:46:57.996095    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93ef078d_6084_421b_9311_c91a4852be8b.slice/crio-6d8a204d2dc946a5040408503e911faf0f3538a66dff6422ff2dd00f7e914b61 WatchSource:0}: Error finding container 6d8a204d2dc946a5040408503e911faf0f3538a66dff6422ff2dd00f7e914b61: Status 404 returned error can't find the container with id 6d8a204d2dc946a5040408503e911faf0f3538a66dff6422ff2dd00f7e914b61
Oct 30 00:46:58.270391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:58.270293    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" event={"ID":"93ef078d-6084-421b-9311-c91a4852be8b","Type":"ContainerStarted","Data":"7392df5c2aa3bc70f467cc7629cde4fa7dd0dea3ae6c8be8e1428d7ee1b0c202"}
Oct 30 00:46:58.270391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:58.270344    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" event={"ID":"93ef078d-6084-421b-9311-c91a4852be8b","Type":"ContainerStarted","Data":"6d8a204d2dc946a5040408503e911faf0f3538a66dff6422ff2dd00f7e914b61"}
Oct 30 00:46:58.307932 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:58.307877    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podStartSLOduration=1.307863886 podStartE2EDuration="1.307863886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:58.306266134 +0000 UTC m=+712.163173889" watchObservedRunningTime="2026-10-30 00:46:58.307863886 +0000 UTC m=+712.164771640"
Oct 30 00:46:58.700618 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:46:58.700584    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:47:00.612585 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:00.612542    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:47:07.707478 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:07.707441    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:47:07.720486 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:07.720461    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:47:08.339828 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:08.339796    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:47:09.952254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:09.952218    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"]
Oct 30 00:47:09.957375 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:09.957347    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:09.975804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:09.975769    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-40ac41\"/\"openshift-service-ca.crt\""
Oct 30 00:47:09.980347 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:09.980310    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-40ac41\"/\"kube-root-ca.crt\""
Oct 30 00:47:09.980593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:09.980568    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-40ac41\"/\"default-dockercfg-5kqq2\""
Oct 30 00:47:09.980694 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:09.980590    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-40ac41\"/\"premium-w2-40ac41-kserve-self-signed-certs\""
Oct 30 00:47:09.988996 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:09.988970    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"]
Oct 30 00:47:09.996058 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:09.996035    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmckk\" (UniqueName: \"kubernetes.io/projected/24858435-0625-4512-bef6-9e9f3a3bb6e2-kube-api-access-tmckk\") pod \"premium-w2-40ac41-kserve-7bdcdb48df-hjpld\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") " pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:09.996168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:09.996088    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-home\") pod \"premium-w2-40ac41-kserve-7bdcdb48df-hjpld\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") " pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:09.996168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:09.996147    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-tmp-dir\") pod \"premium-w2-40ac41-kserve-7bdcdb48df-hjpld\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") " pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:09.996316 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:09.996301    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-model-cache\") pod \"premium-w2-40ac41-kserve-7bdcdb48df-hjpld\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") " pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:09.996352 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:09.996344    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-dshm\") pod \"premium-w2-40ac41-kserve-7bdcdb48df-hjpld\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") " pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:09.996412 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:09.996398    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24858435-0625-4512-bef6-9e9f3a3bb6e2-tls-certs\") pod \"premium-w2-40ac41-kserve-7bdcdb48df-hjpld\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") " pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:10.097391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:10.097356    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmckk\" (UniqueName: \"kubernetes.io/projected/24858435-0625-4512-bef6-9e9f3a3bb6e2-kube-api-access-tmckk\") pod \"premium-w2-40ac41-kserve-7bdcdb48df-hjpld\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") " pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:10.097556 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:10.097415    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-home\") pod \"premium-w2-40ac41-kserve-7bdcdb48df-hjpld\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") " pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:10.097556 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:10.097437    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-tmp-dir\") pod \"premium-w2-40ac41-kserve-7bdcdb48df-hjpld\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") " pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:10.097556 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:10.097514    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-model-cache\") pod \"premium-w2-40ac41-kserve-7bdcdb48df-hjpld\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") " pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:10.097556 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:10.097546    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-dshm\") pod \"premium-w2-40ac41-kserve-7bdcdb48df-hjpld\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") " pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:10.097796 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:10.097567    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24858435-0625-4512-bef6-9e9f3a3bb6e2-tls-certs\") pod \"premium-w2-40ac41-kserve-7bdcdb48df-hjpld\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") " pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:10.097917 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:10.097894    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-home\") pod \"premium-w2-40ac41-kserve-7bdcdb48df-hjpld\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") " pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:10.097984 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:10.097953    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-tmp-dir\") pod \"premium-w2-40ac41-kserve-7bdcdb48df-hjpld\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") " pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:10.098037 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:10.097991    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-model-cache\") pod \"premium-w2-40ac41-kserve-7bdcdb48df-hjpld\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") " pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:10.099878 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:10.099850    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-dshm\") pod \"premium-w2-40ac41-kserve-7bdcdb48df-hjpld\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") " pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:10.100007 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:10.099972    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24858435-0625-4512-bef6-9e9f3a3bb6e2-tls-certs\") pod \"premium-w2-40ac41-kserve-7bdcdb48df-hjpld\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") " pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:10.118313 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:10.118289    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmckk\" (UniqueName: \"kubernetes.io/projected/24858435-0625-4512-bef6-9e9f3a3bb6e2-kube-api-access-tmckk\") pod \"premium-w2-40ac41-kserve-7bdcdb48df-hjpld\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") " pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:10.270096 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:10.270006    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:10.566129 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:10.566103    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"]
Oct 30 00:47:10.568177 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:47:10.568140    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24858435_0625_4512_bef6_9e9f3a3bb6e2.slice/crio-928bfd756666cae39f561f7b67741e2ff3ac7ff396d6aff40ed9318782abf5ac WatchSource:0}: Error finding container 928bfd756666cae39f561f7b67741e2ff3ac7ff396d6aff40ed9318782abf5ac: Status 404 returned error can't find the container with id 928bfd756666cae39f561f7b67741e2ff3ac7ff396d6aff40ed9318782abf5ac
Oct 30 00:47:11.349447 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:11.349411    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" event={"ID":"24858435-0625-4512-bef6-9e9f3a3bb6e2","Type":"ContainerStarted","Data":"3e72a155d8f701e715e3c9edabeb14424014731d3caffcf8415335d317b2c078"}
Oct 30 00:47:11.349846 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:11.349453    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" event={"ID":"24858435-0625-4512-bef6-9e9f3a3bb6e2","Type":"ContainerStarted","Data":"928bfd756666cae39f561f7b67741e2ff3ac7ff396d6aff40ed9318782abf5ac"}
Oct 30 00:47:11.387761 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:11.387698    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podStartSLOduration=2.387684197 podStartE2EDuration="2.387684197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:11.385575472 +0000 UTC m=+725.242483227" watchObservedRunningTime="2026-10-30 00:47:11.387684197 +0000 UTC m=+725.244591951"
Oct 30 00:47:12.270464 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:12.270432    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:16.535168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:16.535130    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:47:16.630675 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:47:16.630640    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:47:20.270585 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.270508    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:20.283484 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.283419    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:20.422390 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.422359    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:47:20.686678 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.686643    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"]
Oct 30 00:47:20.691684 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.691664    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:20.705059 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.705035    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5062f6\"/\"premium-w6-5062f6-kserve-self-signed-certs\""
Oct 30 00:47:20.711072 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.710666    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"]
Oct 30 00:47:20.832546 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.832511    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-home\") pod \"premium-w6-5062f6-kserve-bf6d5d887-2bvlh\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") " pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:20.832739 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.832552    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-model-cache\") pod \"premium-w6-5062f6-kserve-bf6d5d887-2bvlh\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") " pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:20.832739 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.832701    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-tmp-dir\") pod \"premium-w6-5062f6-kserve-bf6d5d887-2bvlh\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") " pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:20.832831 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.832753    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-dshm\") pod \"premium-w6-5062f6-kserve-bf6d5d887-2bvlh\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") " pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:20.832867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.832851    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-tls-certs\") pod \"premium-w6-5062f6-kserve-bf6d5d887-2bvlh\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") " pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:20.832900 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.832879    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd99z\" (UniqueName: \"kubernetes.io/projected/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-kube-api-access-rd99z\") pod \"premium-w6-5062f6-kserve-bf6d5d887-2bvlh\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") " pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:20.933635 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.933600    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-tmp-dir\") pod \"premium-w6-5062f6-kserve-bf6d5d887-2bvlh\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") " pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:20.933856 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.933647    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-dshm\") pod \"premium-w6-5062f6-kserve-bf6d5d887-2bvlh\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") " pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:20.933856 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.933722    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-tls-certs\") pod \"premium-w6-5062f6-kserve-bf6d5d887-2bvlh\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") " pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:20.933856 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.933749    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd99z\" (UniqueName: \"kubernetes.io/projected/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-kube-api-access-rd99z\") pod \"premium-w6-5062f6-kserve-bf6d5d887-2bvlh\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") " pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:20.933856 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.933822    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-home\") pod \"premium-w6-5062f6-kserve-bf6d5d887-2bvlh\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") " pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:20.934080 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.933858    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-model-cache\") pod \"premium-w6-5062f6-kserve-bf6d5d887-2bvlh\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") " pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:20.934330 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.934307    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-model-cache\") pod \"premium-w6-5062f6-kserve-bf6d5d887-2bvlh\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") " pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:20.934474 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.934443    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-tmp-dir\") pod \"premium-w6-5062f6-kserve-bf6d5d887-2bvlh\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") " pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:20.934773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.934541    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-home\") pod \"premium-w6-5062f6-kserve-bf6d5d887-2bvlh\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") " pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:20.936574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.936549    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-dshm\") pod \"premium-w6-5062f6-kserve-bf6d5d887-2bvlh\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") " pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:20.936848 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.936801    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-tls-certs\") pod \"premium-w6-5062f6-kserve-bf6d5d887-2bvlh\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") " pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:20.954536 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:20.954511    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd99z\" (UniqueName: \"kubernetes.io/projected/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-kube-api-access-rd99z\") pod \"premium-w6-5062f6-kserve-bf6d5d887-2bvlh\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") " pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:21.007320 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:21.007293    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:21.304867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:21.304825    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"]
Oct 30 00:47:21.313881 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:47:21.313856    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c7a9b83_c1a2_4460_8bad_5f3967a8d159.slice/crio-77bfb04dbce26b4eadb94e18a50cc920f8d4836310de84fa599233d98abe07d4 WatchSource:0}: Error finding container 77bfb04dbce26b4eadb94e18a50cc920f8d4836310de84fa599233d98abe07d4: Status 404 returned error can't find the container with id 77bfb04dbce26b4eadb94e18a50cc920f8d4836310de84fa599233d98abe07d4
Oct 30 00:47:21.425479 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:21.425437    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" event={"ID":"3c7a9b83-c1a2-4460-8bad-5f3967a8d159","Type":"ContainerStarted","Data":"69980549eeb11bd271e6b6b7cbcf94319e08287456d151e919132bb6a3e6cd65"}
Oct 30 00:47:21.425636 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:21.425488    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" event={"ID":"3c7a9b83-c1a2-4460-8bad-5f3967a8d159","Type":"ContainerStarted","Data":"77bfb04dbce26b4eadb94e18a50cc920f8d4836310de84fa599233d98abe07d4"}
Oct 30 00:47:21.462865 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:21.462820    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podStartSLOduration=1.462806093 podStartE2EDuration="1.462806093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:21.461817343 +0000 UTC m=+735.318725099" watchObservedRunningTime="2026-10-30 00:47:21.462806093 +0000 UTC m=+735.319713847"
Oct 30 00:47:22.008102 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:22.008068    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:27.909933 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:27.909891    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:47:31.007770 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:31.007733    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:31.021243 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:31.021219    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:31.491384 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:31.491352    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:47:41.660820 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.660780    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"]
Oct 30 00:47:41.666495 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.666471    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.691341 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.691316    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-89a676\"/\"default-dockercfg-bpwgn\""
Oct 30 00:47:41.691675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.691564    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-89a676\"/\"premium-w5-89a676-kserve-self-signed-certs\""
Oct 30 00:47:41.691865 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.691614    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-89a676\"/\"kube-root-ca.crt\""
Oct 30 00:47:41.692402 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.692361    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-89a676\"/\"openshift-service-ca.crt\""
Oct 30 00:47:41.698966 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.698938    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"]
Oct 30 00:47:41.782820 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.782783    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-model-cache\") pod \"premium-w5-89a676-kserve-5f558786f4-9lf6q\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") " pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.783008 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.782952    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-dshm\") pod \"premium-w5-89a676-kserve-5f558786f4-9lf6q\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") " pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.783086 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.783051    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-home\") pod \"premium-w5-89a676-kserve-5f558786f4-9lf6q\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") " pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.783152 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.783093    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-tmp-dir\") pod \"premium-w5-89a676-kserve-5f558786f4-9lf6q\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") " pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.783209 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.783157    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n6hn\" (UniqueName: \"kubernetes.io/projected/80fa6588-19e4-4f9a-8291-12775c169de4-kube-api-access-8n6hn\") pod \"premium-w5-89a676-kserve-5f558786f4-9lf6q\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") " pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.783266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.783247    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80fa6588-19e4-4f9a-8291-12775c169de4-tls-certs\") pod \"premium-w5-89a676-kserve-5f558786f4-9lf6q\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") " pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.884511 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.884474    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-dshm\") pod \"premium-w5-89a676-kserve-5f558786f4-9lf6q\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") " pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.884845 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.884814    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-home\") pod \"premium-w5-89a676-kserve-5f558786f4-9lf6q\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") " pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.884980 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.884867    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-tmp-dir\") pod \"premium-w5-89a676-kserve-5f558786f4-9lf6q\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") " pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.884980 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.884928    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n6hn\" (UniqueName: \"kubernetes.io/projected/80fa6588-19e4-4f9a-8291-12775c169de4-kube-api-access-8n6hn\") pod \"premium-w5-89a676-kserve-5f558786f4-9lf6q\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") " pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.885124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.885022    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80fa6588-19e4-4f9a-8291-12775c169de4-tls-certs\") pod \"premium-w5-89a676-kserve-5f558786f4-9lf6q\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") " pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.885124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.885059    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-model-cache\") pod \"premium-w5-89a676-kserve-5f558786f4-9lf6q\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") " pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.885262 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.885230    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-home\") pod \"premium-w5-89a676-kserve-5f558786f4-9lf6q\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") " pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.885405 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.885382    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-model-cache\") pod \"premium-w5-89a676-kserve-5f558786f4-9lf6q\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") " pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.887915 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.885814    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-tmp-dir\") pod \"premium-w5-89a676-kserve-5f558786f4-9lf6q\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") " pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.891809 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.891758    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-dshm\") pod \"premium-w5-89a676-kserve-5f558786f4-9lf6q\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") " pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.894948 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.894902    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80fa6588-19e4-4f9a-8291-12775c169de4-tls-certs\") pod \"premium-w5-89a676-kserve-5f558786f4-9lf6q\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") " pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.930538 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.930413    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n6hn\" (UniqueName: \"kubernetes.io/projected/80fa6588-19e4-4f9a-8291-12775c169de4-kube-api-access-8n6hn\") pod \"premium-w5-89a676-kserve-5f558786f4-9lf6q\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") " pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:41.997931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:41.997891    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:42.305829 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:42.305801    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"]
Oct 30 00:47:42.308218 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:47:42.308184    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80fa6588_19e4_4f9a_8291_12775c169de4.slice/crio-d87f854b47c317f61f821e24dfcc30c6836d0e9993088b58762d44ce0076c022 WatchSource:0}: Error finding container d87f854b47c317f61f821e24dfcc30c6836d0e9993088b58762d44ce0076c022: Status 404 returned error can't find the container with id d87f854b47c317f61f821e24dfcc30c6836d0e9993088b58762d44ce0076c022
Oct 30 00:47:42.558929 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:42.558553    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" event={"ID":"80fa6588-19e4-4f9a-8291-12775c169de4","Type":"ContainerStarted","Data":"37f62e9ea5065c3d44e742f3ab0b92bde8bf4924a7960b1121b7ab8c9a6e39d3"}
Oct 30 00:47:42.558929 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:42.558609    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" event={"ID":"80fa6588-19e4-4f9a-8291-12775c169de4","Type":"ContainerStarted","Data":"d87f854b47c317f61f821e24dfcc30c6836d0e9993088b58762d44ce0076c022"}
Oct 30 00:47:42.595257 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:42.595180    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podStartSLOduration=1.595166523 podStartE2EDuration="1.595166523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:42.594638508 +0000 UTC m=+756.451546263" watchObservedRunningTime="2026-10-30 00:47:42.595166523 +0000 UTC m=+756.452074277"
Oct 30 00:47:42.998983 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:42.998874    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:45.329661 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.329625    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr"]
Oct 30 00:47:45.334050 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.334025    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr"
Oct 30 00:47:45.346592 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.346567    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:47:45.356585 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.356562    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:47:45.356690 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.356599    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:47:45.356690 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.356618    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:47:45.358675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.358653    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr"]
Oct 30 00:47:45.429041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.429007    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03da01dc-kkmhr\" (UID: \"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr"
Oct 30 00:47:45.429186 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.429052    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scjhp\" (UniqueName: \"kubernetes.io/projected/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-kube-api-access-scjhp\") pod \"maas-api-revoke-sub-03da01dc-kkmhr\" (UID: \"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr"
Oct 30 00:47:45.429186 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.429137    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03da01dc-kkmhr\" (UID: \"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr"
Oct 30 00:47:45.529860 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.529832    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03da01dc-kkmhr\" (UID: \"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr"
Oct 30 00:47:45.529987 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.529870    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scjhp\" (UniqueName: \"kubernetes.io/projected/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-kube-api-access-scjhp\") pod \"maas-api-revoke-sub-03da01dc-kkmhr\" (UID: \"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr"
Oct 30 00:47:45.529987 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.529891    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03da01dc-kkmhr\" (UID: \"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr"
Oct 30 00:47:45.530576 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.530556    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03da01dc-kkmhr\" (UID: \"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr"
Oct 30 00:47:45.532157 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.532141    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03da01dc-kkmhr\" (UID: \"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr"
Oct 30 00:47:45.550464 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.550440    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scjhp\" (UniqueName: \"kubernetes.io/projected/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-kube-api-access-scjhp\") pod \"maas-api-revoke-sub-03da01dc-kkmhr\" (UID: \"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr"
Oct 30 00:47:45.648025 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.647968    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr"
Oct 30 00:47:45.859114 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.859057    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"]
Oct 30 00:47:45.864405 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.864380    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:45.878116 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.878093    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2b4105\"/\"kube-root-ca.crt\""
Oct 30 00:47:45.878226 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.878160    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2b4105\"/\"openshift-service-ca.crt\""
Oct 30 00:47:45.888022 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.887993    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2b4105\"/\"premium-w0-2b4105-kserve-self-signed-certs\""
Oct 30 00:47:45.888022 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.888008    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2b4105\"/\"default-dockercfg-wmkj4\""
Oct 30 00:47:45.890387 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.890361    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"]
Oct 30 00:47:45.934424 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.934363    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxfw7\" (UniqueName: \"kubernetes.io/projected/ef896512-2f16-4fd1-a732-cf9c232994e0-kube-api-access-vxfw7\") pod \"premium-w0-2b4105-kserve-6f6b659d9d-99vzw\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") " pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:45.934566 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.934494    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-dshm\") pod \"premium-w0-2b4105-kserve-6f6b659d9d-99vzw\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") " pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:45.934608 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.934563    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-tmp-dir\") pod \"premium-w0-2b4105-kserve-6f6b659d9d-99vzw\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") " pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:45.934608 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.934592    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef896512-2f16-4fd1-a732-cf9c232994e0-tls-certs\") pod \"premium-w0-2b4105-kserve-6f6b659d9d-99vzw\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") " pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:45.934673 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.934615    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-model-cache\") pod \"premium-w0-2b4105-kserve-6f6b659d9d-99vzw\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") " pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:45.934673 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.934653    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-home\") pod \"premium-w0-2b4105-kserve-6f6b659d9d-99vzw\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") " pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:45.943194 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:45.941779    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr"]
Oct 30 00:47:45.944101 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:47:45.944078    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7f8ba0_bbd4_4a2b_9cdc_fb763111f4b0.slice/crio-08229dc259edae2e53c2467d98f882beb0335f8a8a70966a18e57a846a4e2578 WatchSource:0}: Error finding container 08229dc259edae2e53c2467d98f882beb0335f8a8a70966a18e57a846a4e2578: Status 404 returned error can't find the container with id 08229dc259edae2e53c2467d98f882beb0335f8a8a70966a18e57a846a4e2578
Oct 30 00:47:46.035329 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.035299    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxfw7\" (UniqueName: \"kubernetes.io/projected/ef896512-2f16-4fd1-a732-cf9c232994e0-kube-api-access-vxfw7\") pod \"premium-w0-2b4105-kserve-6f6b659d9d-99vzw\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") " pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:46.035441 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.035430    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-dshm\") pod \"premium-w0-2b4105-kserve-6f6b659d9d-99vzw\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") " pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:46.035487 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.035472    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-tmp-dir\") pod \"premium-w0-2b4105-kserve-6f6b659d9d-99vzw\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") " pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:46.035537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.035499    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef896512-2f16-4fd1-a732-cf9c232994e0-tls-certs\") pod \"premium-w0-2b4105-kserve-6f6b659d9d-99vzw\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") " pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:46.035537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.035529    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-model-cache\") pod \"premium-w0-2b4105-kserve-6f6b659d9d-99vzw\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") " pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:46.035626 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.035574    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-home\") pod \"premium-w0-2b4105-kserve-6f6b659d9d-99vzw\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") " pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:46.035933 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.035902    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-tmp-dir\") pod \"premium-w0-2b4105-kserve-6f6b659d9d-99vzw\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") " pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:46.036042 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.035938    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-model-cache\") pod \"premium-w0-2b4105-kserve-6f6b659d9d-99vzw\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") " pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:46.036042 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.035951    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-home\") pod \"premium-w0-2b4105-kserve-6f6b659d9d-99vzw\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") " pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:46.037618 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.037594    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-dshm\") pod \"premium-w0-2b4105-kserve-6f6b659d9d-99vzw\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") " pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:46.037871 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.037857    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef896512-2f16-4fd1-a732-cf9c232994e0-tls-certs\") pod \"premium-w0-2b4105-kserve-6f6b659d9d-99vzw\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") " pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:46.055757 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.055739    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxfw7\" (UniqueName: \"kubernetes.io/projected/ef896512-2f16-4fd1-a732-cf9c232994e0-kube-api-access-vxfw7\") pod \"premium-w0-2b4105-kserve-6f6b659d9d-99vzw\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") " pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:46.178255 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.178228    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:46.476640 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.476612    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"]
Oct 30 00:47:46.478788 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:47:46.478756    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef896512_2f16_4fd1_a732_cf9c232994e0.slice/crio-ae0de0e10fced29a01287b6a240aaff5e0e13b11f200e1d4c75e12ef95e43d80 WatchSource:0}: Error finding container ae0de0e10fced29a01287b6a240aaff5e0e13b11f200e1d4c75e12ef95e43d80: Status 404 returned error can't find the container with id ae0de0e10fced29a01287b6a240aaff5e0e13b11f200e1d4c75e12ef95e43d80
Oct 30 00:47:46.588553 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.588519    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr" event={"ID":"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0","Type":"ContainerStarted","Data":"dd1dcde3c069917f7d8e09267dc913314057f20512efd3b346c934a52f63a6c2"}
Oct 30 00:47:46.588751 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.588578    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr" event={"ID":"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0","Type":"ContainerStarted","Data":"08229dc259edae2e53c2467d98f882beb0335f8a8a70966a18e57a846a4e2578"}
Oct 30 00:47:46.590497 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.590466    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" event={"ID":"ef896512-2f16-4fd1-a732-cf9c232994e0","Type":"ContainerStarted","Data":"438d0e365769ef86eb391a1c4a9580cf527413e309ff1f5d9238af1eeb9ff251"}
Oct 30 00:47:46.590497 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.590507    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" event={"ID":"ef896512-2f16-4fd1-a732-cf9c232994e0","Type":"ContainerStarted","Data":"ae0de0e10fced29a01287b6a240aaff5e0e13b11f200e1d4c75e12ef95e43d80"}
Oct 30 00:47:46.626897 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.626655    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr" podStartSLOduration=1.626637925 podStartE2EDuration="1.626637925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:46.623865111 +0000 UTC m=+760.480772867" watchObservedRunningTime="2026-10-30 00:47:46.626637925 +0000 UTC m=+760.483545682"
Oct 30 00:47:46.630197 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:47:46.630169    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:47:46.660287 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:46.660228    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podStartSLOduration=1.660211535 podStartE2EDuration="1.660211535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:46.6595137 +0000 UTC m=+760.516421455" watchObservedRunningTime="2026-10-30 00:47:46.660211535 +0000 UTC m=+760.517119291"
Oct 30 00:47:47.178461 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:47.178382    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:47.265505 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:47.265468    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:47:47.608065 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:47.607052    6099 generic.go:356] "Generic (PLEG): container finished" podID="fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0" containerID="dd1dcde3c069917f7d8e09267dc913314057f20512efd3b346c934a52f63a6c2" exitCode=0
Oct 30 00:47:47.608065 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:47.607551    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr" event={"ID":"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0","Type":"ContainerDied","Data":"dd1dcde3c069917f7d8e09267dc913314057f20512efd3b346c934a52f63a6c2"}
Oct 30 00:47:49.150779 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:49.146906    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr"
Oct 30 00:47:49.334751 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:49.334696    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-kube-api-access-scjhp\" (UniqueName: \"kubernetes.io/projected/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-kube-api-access-scjhp\") pod \"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0\" (UID: \"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0\") "
Oct 30 00:47:49.334943 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:49.334763    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-maas-api-xks-ca\") pod \"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0\" (UID: \"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0\") "
Oct 30 00:47:49.334943 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:49.334814    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-maas-api-openshift-ca\") pod \"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0\" (UID: \"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0\") "
Oct 30 00:47:49.335175 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:49.335148    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-maas-api-openshift-ca" pod "fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0" (UID: "fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:47:49.335672 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:49.335638    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:47:49.337657 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:49.337628    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-maas-api-xks-ca" pod "fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0" (UID: "fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:47:49.337786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:49.337665    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-kube-api-access-scjhp" pod "fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0" (UID: "fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0"). InnerVolumeSpecName "kube-api-access-scjhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:47:49.437329 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:49.437287    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scjhp\" (UniqueName: \"kubernetes.io/projected/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-kube-api-access-scjhp\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:47:49.437329 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:49.437323    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:47:49.638755 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:49.638668    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr" event={"ID":"fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0","Type":"ContainerDied","Data":"08229dc259edae2e53c2467d98f882beb0335f8a8a70966a18e57a846a4e2578"}
Oct 30 00:47:49.638755 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:49.638739    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08229dc259edae2e53c2467d98f882beb0335f8a8a70966a18e57a846a4e2578"
Oct 30 00:47:49.638755 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:49.638741    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr"
Oct 30 00:47:51.998824 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:51.998793    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:52.011932 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.011908    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:52.665904 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.665871    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:47:52.680963 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.680931    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c"]
Oct 30 00:47:52.681427 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.681412    6099 cpu_manager.go:414] "containerMap: removing container" podUID="fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0" containerName="revoke-keys"
Oct 30 00:47:52.681477 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.681429    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0" containerName="revoke-keys"
Oct 30 00:47:52.681552 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.681543    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0" containerName="revoke-keys"
Oct 30 00:47:52.684945 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.684926    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c"
Oct 30 00:47:52.697566 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.697538    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:47:52.697670 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.697590    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:47:52.697670 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.697625    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:47:52.697788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.697678    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:47:52.703421 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.703401    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c"]
Oct 30 00:47:52.775583 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.775552    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hph5v\" (UniqueName: \"kubernetes.io/projected/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-kube-api-access-hph5v\") pod \"maas-api-revoke-sub-0726217b-8nd6c\" (UID: \"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c"
Oct 30 00:47:52.775861 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.775699    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0726217b-8nd6c\" (UID: \"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c"
Oct 30 00:47:52.775861 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.775778    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0726217b-8nd6c\" (UID: \"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c"
Oct 30 00:47:52.876885 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.876854    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hph5v\" (UniqueName: \"kubernetes.io/projected/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-kube-api-access-hph5v\") pod \"maas-api-revoke-sub-0726217b-8nd6c\" (UID: \"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c"
Oct 30 00:47:52.877035 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.876959    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0726217b-8nd6c\" (UID: \"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c"
Oct 30 00:47:52.877035 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.876983    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0726217b-8nd6c\" (UID: \"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c"
Oct 30 00:47:52.877606 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.877583    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0726217b-8nd6c\" (UID: \"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c"
Oct 30 00:47:52.879286 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.879268    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0726217b-8nd6c\" (UID: \"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c"
Oct 30 00:47:52.897615 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.897592    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hph5v\" (UniqueName: \"kubernetes.io/projected/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-kube-api-access-hph5v\") pod \"maas-api-revoke-sub-0726217b-8nd6c\" (UID: \"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c"
Oct 30 00:47:52.999739 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:52.999662    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c"
Oct 30 00:47:53.298629 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:53.298605    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c"]
Oct 30 00:47:53.301265 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:47:53.301235    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee9d15a_a18e_44b7_8b16_2e30fd8ce423.slice/crio-906d78aa51e906e4225f65f212d261762543c9c55973fafef2d85e71110603db WatchSource:0}: Error finding container 906d78aa51e906e4225f65f212d261762543c9c55973fafef2d85e71110603db: Status 404 returned error can't find the container with id 906d78aa51e906e4225f65f212d261762543c9c55973fafef2d85e71110603db
Oct 30 00:47:53.663068 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:53.663029    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c" event={"ID":"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423","Type":"ContainerStarted","Data":"7513319f9ee01cccb7a3179c08b1a765334a874466df6c5fdd7e2bc8cf47e297"}
Oct 30 00:47:53.663244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:53.663092    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c" event={"ID":"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423","Type":"ContainerStarted","Data":"906d78aa51e906e4225f65f212d261762543c9c55973fafef2d85e71110603db"}
Oct 30 00:47:54.670821 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:54.670785    6099 generic.go:356] "Generic (PLEG): container finished" podID="1ee9d15a-a18e-44b7-8b16-2e30fd8ce423" containerID="7513319f9ee01cccb7a3179c08b1a765334a874466df6c5fdd7e2bc8cf47e297" exitCode=0
Oct 30 00:47:54.671190 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:54.670840    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c" event={"ID":"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423","Type":"ContainerDied","Data":"7513319f9ee01cccb7a3179c08b1a765334a874466df6c5fdd7e2bc8cf47e297"}
Oct 30 00:47:55.831993 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:55.831970    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c"
Oct 30 00:47:55.907977 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:55.907950    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-maas-api-xks-ca\") pod \"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423\" (UID: \"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423\") "
Oct 30 00:47:55.908137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:55.908015    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-maas-api-openshift-ca\") pod \"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423\" (UID: \"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423\") "
Oct 30 00:47:55.908137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:55.908055    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-kube-api-access-hph5v\" (UniqueName: \"kubernetes.io/projected/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-kube-api-access-hph5v\") pod \"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423\" (UID: \"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423\") "
Oct 30 00:47:55.908421 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:55.908396    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-maas-api-openshift-ca" pod "1ee9d15a-a18e-44b7-8b16-2e30fd8ce423" (UID: "1ee9d15a-a18e-44b7-8b16-2e30fd8ce423"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:47:55.910210 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:55.910184    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-kube-api-access-hph5v" pod "1ee9d15a-a18e-44b7-8b16-2e30fd8ce423" (UID: "1ee9d15a-a18e-44b7-8b16-2e30fd8ce423"). InnerVolumeSpecName "kube-api-access-hph5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:47:55.910321 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:55.910283    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-maas-api-xks-ca" pod "1ee9d15a-a18e-44b7-8b16-2e30fd8ce423" (UID: "1ee9d15a-a18e-44b7-8b16-2e30fd8ce423"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:47:56.009397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:56.009335    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:47:56.009397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:56.009360    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:47:56.009397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:56.009369    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hph5v\" (UniqueName: \"kubernetes.io/projected/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423-kube-api-access-hph5v\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:47:56.179265 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:56.179217    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:56.191692 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:56.191661    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:56.689202 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:56.689175    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c"
Oct 30 00:47:56.689372 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:56.689214    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c" event={"ID":"1ee9d15a-a18e-44b7-8b16-2e30fd8ce423","Type":"ContainerDied","Data":"906d78aa51e906e4225f65f212d261762543c9c55973fafef2d85e71110603db"}
Oct 30 00:47:56.689372 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:56.689281    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="906d78aa51e906e4225f65f212d261762543c9c55973fafef2d85e71110603db"
Oct 30 00:47:56.700723 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:56.700689    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:47:59.906351 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:59.906314    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc"]
Oct 30 00:47:59.906896 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:59.906882    6099 cpu_manager.go:414] "containerMap: removing container" podUID="1ee9d15a-a18e-44b7-8b16-2e30fd8ce423" containerName="revoke-keys"
Oct 30 00:47:59.906944 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:59.906898    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee9d15a-a18e-44b7-8b16-2e30fd8ce423" containerName="revoke-keys"
Oct 30 00:47:59.906987 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:59.906978    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee9d15a-a18e-44b7-8b16-2e30fd8ce423" containerName="revoke-keys"
Oct 30 00:47:59.911547 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:59.911531    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc"
Oct 30 00:47:59.931376 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:59.931337    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:47:59.941315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:59.941286    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:47:59.941491 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:59.941433    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:47:59.941591 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:59.941456    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:47:59.944304 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:59.944280    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc"]
Oct 30 00:47:59.953259 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:59.953239    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/597d5e40-34d9-4362-acb6-447fa9664d80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf7b88e8-9v8bc\" (UID: \"597d5e40-34d9-4362-acb6-447fa9664d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc"
Oct 30 00:47:59.953338 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:59.953317    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/597d5e40-34d9-4362-acb6-447fa9664d80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf7b88e8-9v8bc\" (UID: \"597d5e40-34d9-4362-acb6-447fa9664d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc"
Oct 30 00:47:59.953453 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:47:59.953436    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxb4p\" (UniqueName: \"kubernetes.io/projected/597d5e40-34d9-4362-acb6-447fa9664d80-kube-api-access-sxb4p\") pod \"maas-api-revoke-sub-cf7b88e8-9v8bc\" (UID: \"597d5e40-34d9-4362-acb6-447fa9664d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc"
Oct 30 00:48:00.055127 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:00.055088    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/597d5e40-34d9-4362-acb6-447fa9664d80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf7b88e8-9v8bc\" (UID: \"597d5e40-34d9-4362-acb6-447fa9664d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc"
Oct 30 00:48:00.055290 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:00.055139    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/597d5e40-34d9-4362-acb6-447fa9664d80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf7b88e8-9v8bc\" (UID: \"597d5e40-34d9-4362-acb6-447fa9664d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc"
Oct 30 00:48:00.055290 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:00.055226    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxb4p\" (UniqueName: \"kubernetes.io/projected/597d5e40-34d9-4362-acb6-447fa9664d80-kube-api-access-sxb4p\") pod \"maas-api-revoke-sub-cf7b88e8-9v8bc\" (UID: \"597d5e40-34d9-4362-acb6-447fa9664d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc"
Oct 30 00:48:00.055752 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:00.055700    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/597d5e40-34d9-4362-acb6-447fa9664d80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf7b88e8-9v8bc\" (UID: \"597d5e40-34d9-4362-acb6-447fa9664d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc"
Oct 30 00:48:00.057450 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:00.057430    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/597d5e40-34d9-4362-acb6-447fa9664d80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf7b88e8-9v8bc\" (UID: \"597d5e40-34d9-4362-acb6-447fa9664d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc"
Oct 30 00:48:00.076342 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:00.076317    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxb4p\" (UniqueName: \"kubernetes.io/projected/597d5e40-34d9-4362-acb6-447fa9664d80-kube-api-access-sxb4p\") pod \"maas-api-revoke-sub-cf7b88e8-9v8bc\" (UID: \"597d5e40-34d9-4362-acb6-447fa9664d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc"
Oct 30 00:48:00.233551 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:00.233470    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc"
Oct 30 00:48:00.535371 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:00.535342    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc"]
Oct 30 00:48:00.538234 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:48:00.538205    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597d5e40_34d9_4362_acb6_447fa9664d80.slice/crio-5985ef9f8fc4b2a50218c2621a4add91ac0e12a379665ee49fa1fca8d4630c7d WatchSource:0}: Error finding container 5985ef9f8fc4b2a50218c2621a4add91ac0e12a379665ee49fa1fca8d4630c7d: Status 404 returned error can't find the container with id 5985ef9f8fc4b2a50218c2621a4add91ac0e12a379665ee49fa1fca8d4630c7d
Oct 30 00:48:00.713216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:00.713183    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc" event={"ID":"597d5e40-34d9-4362-acb6-447fa9664d80","Type":"ContainerStarted","Data":"5985ef9f8fc4b2a50218c2621a4add91ac0e12a379665ee49fa1fca8d4630c7d"}
Oct 30 00:48:01.723200 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:01.723119    6099 generic.go:356] "Generic (PLEG): container finished" podID="597d5e40-34d9-4362-acb6-447fa9664d80" containerID="c7385e4f2e7ae265292b256134aac379bad047675a5c9bf01c81ba2914b6e2c7" exitCode=0
Oct 30 00:48:01.723586 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:01.723204    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc" event={"ID":"597d5e40-34d9-4362-acb6-447fa9664d80","Type":"ContainerDied","Data":"c7385e4f2e7ae265292b256134aac379bad047675a5c9bf01c81ba2914b6e2c7"}
Oct 30 00:48:02.883131 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:02.883106    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc"
Oct 30 00:48:02.993232 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:02.993147    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/597d5e40-34d9-4362-acb6-447fa9664d80-kube-api-access-sxb4p\" (UniqueName: \"kubernetes.io/projected/597d5e40-34d9-4362-acb6-447fa9664d80-kube-api-access-sxb4p\") pod \"597d5e40-34d9-4362-acb6-447fa9664d80\" (UID: \"597d5e40-34d9-4362-acb6-447fa9664d80\") "
Oct 30 00:48:02.993402 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:02.993256    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/597d5e40-34d9-4362-acb6-447fa9664d80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/597d5e40-34d9-4362-acb6-447fa9664d80-maas-api-openshift-ca\") pod \"597d5e40-34d9-4362-acb6-447fa9664d80\" (UID: \"597d5e40-34d9-4362-acb6-447fa9664d80\") "
Oct 30 00:48:02.993402 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:02.993326    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/597d5e40-34d9-4362-acb6-447fa9664d80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/597d5e40-34d9-4362-acb6-447fa9664d80-maas-api-xks-ca\") pod \"597d5e40-34d9-4362-acb6-447fa9664d80\" (UID: \"597d5e40-34d9-4362-acb6-447fa9664d80\") "
Oct 30 00:48:02.993588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:02.993563    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/597d5e40-34d9-4362-acb6-447fa9664d80-maas-api-openshift-ca" pod "597d5e40-34d9-4362-acb6-447fa9664d80" (UID: "597d5e40-34d9-4362-acb6-447fa9664d80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:48:02.993834 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:02.993816    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/597d5e40-34d9-4362-acb6-447fa9664d80-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:02.995186 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:02.995162    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/597d5e40-34d9-4362-acb6-447fa9664d80-kube-api-access-sxb4p" pod "597d5e40-34d9-4362-acb6-447fa9664d80" (UID: "597d5e40-34d9-4362-acb6-447fa9664d80"). InnerVolumeSpecName "kube-api-access-sxb4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:02.995288 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:02.995255    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/597d5e40-34d9-4362-acb6-447fa9664d80-maas-api-xks-ca" pod "597d5e40-34d9-4362-acb6-447fa9664d80" (UID: "597d5e40-34d9-4362-acb6-447fa9664d80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:03.094922 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:03.094890    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxb4p\" (UniqueName: \"kubernetes.io/projected/597d5e40-34d9-4362-acb6-447fa9664d80-kube-api-access-sxb4p\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:03.094922 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:03.094917    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/597d5e40-34d9-4362-acb6-447fa9664d80-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:03.736823 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:03.736790    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc" event={"ID":"597d5e40-34d9-4362-acb6-447fa9664d80","Type":"ContainerDied","Data":"5985ef9f8fc4b2a50218c2621a4add91ac0e12a379665ee49fa1fca8d4630c7d"}
Oct 30 00:48:03.736823 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:03.736822    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc"
Oct 30 00:48:03.737032 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:03.736837    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5985ef9f8fc4b2a50218c2621a4add91ac0e12a379665ee49fa1fca8d4630c7d"
Oct 30 00:48:08.275207 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.275169    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5"]
Oct 30 00:48:08.275920 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.275901    6099 cpu_manager.go:414] "containerMap: removing container" podUID="597d5e40-34d9-4362-acb6-447fa9664d80" containerName="revoke-keys"
Oct 30 00:48:08.275976 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.275921    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="597d5e40-34d9-4362-acb6-447fa9664d80" containerName="revoke-keys"
Oct 30 00:48:08.276069 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.276059    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="597d5e40-34d9-4362-acb6-447fa9664d80" containerName="revoke-keys"
Oct 30 00:48:08.279650 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.279628    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5"
Oct 30 00:48:08.292192 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.292165    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:48:08.292192 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.292180    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:48:08.302144 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.302123    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:48:08.302277 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.302225    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:48:08.304951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.304928    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5"]
Oct 30 00:48:08.368676 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.368636    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb248\" (UniqueName: \"kubernetes.io/projected/51921f0c-2f68-4220-ba0d-c95ae82ad03f-kube-api-access-wb248\") pod \"maas-api-revoke-sub-fb9196ee-4j7j5\" (UID: \"51921f0c-2f68-4220-ba0d-c95ae82ad03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5"
Oct 30 00:48:08.368676 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.368679    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51921f0c-2f68-4220-ba0d-c95ae82ad03f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb9196ee-4j7j5\" (UID: \"51921f0c-2f68-4220-ba0d-c95ae82ad03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5"
Oct 30 00:48:08.368967 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.368945    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51921f0c-2f68-4220-ba0d-c95ae82ad03f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb9196ee-4j7j5\" (UID: \"51921f0c-2f68-4220-ba0d-c95ae82ad03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5"
Oct 30 00:48:08.469832 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.469797    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51921f0c-2f68-4220-ba0d-c95ae82ad03f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb9196ee-4j7j5\" (UID: \"51921f0c-2f68-4220-ba0d-c95ae82ad03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5"
Oct 30 00:48:08.469997 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.469845    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb248\" (UniqueName: \"kubernetes.io/projected/51921f0c-2f68-4220-ba0d-c95ae82ad03f-kube-api-access-wb248\") pod \"maas-api-revoke-sub-fb9196ee-4j7j5\" (UID: \"51921f0c-2f68-4220-ba0d-c95ae82ad03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5"
Oct 30 00:48:08.469997 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.469865    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51921f0c-2f68-4220-ba0d-c95ae82ad03f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb9196ee-4j7j5\" (UID: \"51921f0c-2f68-4220-ba0d-c95ae82ad03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5"
Oct 30 00:48:08.470526 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.470497    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51921f0c-2f68-4220-ba0d-c95ae82ad03f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb9196ee-4j7j5\" (UID: \"51921f0c-2f68-4220-ba0d-c95ae82ad03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5"
Oct 30 00:48:08.472217 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.472196    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51921f0c-2f68-4220-ba0d-c95ae82ad03f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb9196ee-4j7j5\" (UID: \"51921f0c-2f68-4220-ba0d-c95ae82ad03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5"
Oct 30 00:48:08.490556 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.490533    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb248\" (UniqueName: \"kubernetes.io/projected/51921f0c-2f68-4220-ba0d-c95ae82ad03f-kube-api-access-wb248\") pod \"maas-api-revoke-sub-fb9196ee-4j7j5\" (UID: \"51921f0c-2f68-4220-ba0d-c95ae82ad03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5"
Oct 30 00:48:08.592504 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.592471    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs"]
Oct 30 00:48:08.592840 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.592820    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5"
Oct 30 00:48:08.596272 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.596258    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs"
Oct 30 00:48:08.615030 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.614999    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs"]
Oct 30 00:48:08.679490 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.679458    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ef64c63-lltcs\" (UID: \"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs"
Oct 30 00:48:08.679648 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.679619    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ef64c63-lltcs\" (UID: \"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs"
Oct 30 00:48:08.679759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.679744    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhsxf\" (UniqueName: \"kubernetes.io/projected/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-kube-api-access-dhsxf\") pod \"maas-api-revoke-sub-5ef64c63-lltcs\" (UID: \"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs"
Oct 30 00:48:08.781054 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.781016    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ef64c63-lltcs\" (UID: \"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs"
Oct 30 00:48:08.781236 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.781098    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ef64c63-lltcs\" (UID: \"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs"
Oct 30 00:48:08.781236 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.781143    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhsxf\" (UniqueName: \"kubernetes.io/projected/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-kube-api-access-dhsxf\") pod \"maas-api-revoke-sub-5ef64c63-lltcs\" (UID: \"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs"
Oct 30 00:48:08.781776 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.781754    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ef64c63-lltcs\" (UID: \"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs"
Oct 30 00:48:08.783438 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.783414    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ef64c63-lltcs\" (UID: \"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs"
Oct 30 00:48:08.801977 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.801951    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhsxf\" (UniqueName: \"kubernetes.io/projected/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-kube-api-access-dhsxf\") pod \"maas-api-revoke-sub-5ef64c63-lltcs\" (UID: \"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs"
Oct 30 00:48:08.898499 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.898173    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5"]
Oct 30 00:48:08.900668 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:48:08.900634    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51921f0c_2f68_4220_ba0d_c95ae82ad03f.slice/crio-14c420253a647262a35904ba0b380bdf8bed3b992c7c3f7cd7342256b4d40c04 WatchSource:0}: Error finding container 14c420253a647262a35904ba0b380bdf8bed3b992c7c3f7cd7342256b4d40c04: Status 404 returned error can't find the container with id 14c420253a647262a35904ba0b380bdf8bed3b992c7c3f7cd7342256b4d40c04
Oct 30 00:48:08.935576 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:08.935547    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs"
Oct 30 00:48:09.235221 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:09.235171    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs"]
Oct 30 00:48:09.237625 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:48:09.237597    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d1b67c_e24b_4b75_bf55_c2c7c7d57285.slice/crio-e3a8942a9ed32d4fd8c45c0287a829080a96e2da0ff7099d144feb5bb332f0db WatchSource:0}: Error finding container e3a8942a9ed32d4fd8c45c0287a829080a96e2da0ff7099d144feb5bb332f0db: Status 404 returned error can't find the container with id e3a8942a9ed32d4fd8c45c0287a829080a96e2da0ff7099d144feb5bb332f0db
Oct 30 00:48:09.876887 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:09.876848    6099 generic.go:356] "Generic (PLEG): container finished" podID="51921f0c-2f68-4220-ba0d-c95ae82ad03f" containerID="794035a630a453f002c1efb4d98cd06f10ae5a2625e39525bbb8b36af3d8bb39" exitCode=0
Oct 30 00:48:09.877349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:09.876940    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5" event={"ID":"51921f0c-2f68-4220-ba0d-c95ae82ad03f","Type":"ContainerDied","Data":"794035a630a453f002c1efb4d98cd06f10ae5a2625e39525bbb8b36af3d8bb39"}
Oct 30 00:48:09.877349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:09.876986    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5" event={"ID":"51921f0c-2f68-4220-ba0d-c95ae82ad03f","Type":"ContainerStarted","Data":"14c420253a647262a35904ba0b380bdf8bed3b992c7c3f7cd7342256b4d40c04"}
Oct 30 00:48:09.878515 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:09.878489    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs" event={"ID":"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285","Type":"ContainerStarted","Data":"2ad9b9848f85656b5068b046e6e9b2260d971dd7e683d9fbae6dc9f10b426968"}
Oct 30 00:48:09.878612 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:09.878534    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs" event={"ID":"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285","Type":"ContainerStarted","Data":"e3a8942a9ed32d4fd8c45c0287a829080a96e2da0ff7099d144feb5bb332f0db"}
Oct 30 00:48:09.949812 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:09.949754    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs" podStartSLOduration=1.949739809 podStartE2EDuration="1.949739809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:09.948034455 +0000 UTC m=+783.804942210" watchObservedRunningTime="2026-10-30 00:48:09.949739809 +0000 UTC m=+783.806647564"
Oct 30 00:48:10.885275 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:10.885241    6099 generic.go:356] "Generic (PLEG): container finished" podID="c5d1b67c-e24b-4b75-bf55-c2c7c7d57285" containerID="2ad9b9848f85656b5068b046e6e9b2260d971dd7e683d9fbae6dc9f10b426968" exitCode=0
Oct 30 00:48:10.885687 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:10.885308    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs" event={"ID":"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285","Type":"ContainerDied","Data":"2ad9b9848f85656b5068b046e6e9b2260d971dd7e683d9fbae6dc9f10b426968"}
Oct 30 00:48:11.049126 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:11.049092    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5"
Oct 30 00:48:11.108336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:11.108305    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51921f0c-2f68-4220-ba0d-c95ae82ad03f-kube-api-access-wb248\" (UniqueName: \"kubernetes.io/projected/51921f0c-2f68-4220-ba0d-c95ae82ad03f-kube-api-access-wb248\") pod \"51921f0c-2f68-4220-ba0d-c95ae82ad03f\" (UID: \"51921f0c-2f68-4220-ba0d-c95ae82ad03f\") "
Oct 30 00:48:11.108491 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:11.108368    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51921f0c-2f68-4220-ba0d-c95ae82ad03f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51921f0c-2f68-4220-ba0d-c95ae82ad03f-maas-api-openshift-ca\") pod \"51921f0c-2f68-4220-ba0d-c95ae82ad03f\" (UID: \"51921f0c-2f68-4220-ba0d-c95ae82ad03f\") "
Oct 30 00:48:11.108491 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:11.108461    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51921f0c-2f68-4220-ba0d-c95ae82ad03f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51921f0c-2f68-4220-ba0d-c95ae82ad03f-maas-api-xks-ca\") pod \"51921f0c-2f68-4220-ba0d-c95ae82ad03f\" (UID: \"51921f0c-2f68-4220-ba0d-c95ae82ad03f\") "
Oct 30 00:48:11.108847 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:11.108814    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51921f0c-2f68-4220-ba0d-c95ae82ad03f-maas-api-openshift-ca" pod "51921f0c-2f68-4220-ba0d-c95ae82ad03f" (UID: "51921f0c-2f68-4220-ba0d-c95ae82ad03f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:48:11.109032 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:11.109013    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51921f0c-2f68-4220-ba0d-c95ae82ad03f-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:11.110538 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:11.110509    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51921f0c-2f68-4220-ba0d-c95ae82ad03f-kube-api-access-wb248" pod "51921f0c-2f68-4220-ba0d-c95ae82ad03f" (UID: "51921f0c-2f68-4220-ba0d-c95ae82ad03f"). InnerVolumeSpecName "kube-api-access-wb248". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:11.110954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:11.110934    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51921f0c-2f68-4220-ba0d-c95ae82ad03f-maas-api-xks-ca" pod "51921f0c-2f68-4220-ba0d-c95ae82ad03f" (UID: "51921f0c-2f68-4220-ba0d-c95ae82ad03f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:11.210121 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:11.210043    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb248\" (UniqueName: \"kubernetes.io/projected/51921f0c-2f68-4220-ba0d-c95ae82ad03f-kube-api-access-wb248\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:11.210121 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:11.210076    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51921f0c-2f68-4220-ba0d-c95ae82ad03f-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:11.892526 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:11.892500    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5"
Oct 30 00:48:11.892940 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:11.892534    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5" event={"ID":"51921f0c-2f68-4220-ba0d-c95ae82ad03f","Type":"ContainerDied","Data":"14c420253a647262a35904ba0b380bdf8bed3b992c7c3f7cd7342256b4d40c04"}
Oct 30 00:48:11.892940 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:11.892580    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14c420253a647262a35904ba0b380bdf8bed3b992c7c3f7cd7342256b4d40c04"
Oct 30 00:48:12.035265 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:12.035239    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs"
Oct 30 00:48:12.120696 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:12.120663    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-maas-api-xks-ca\") pod \"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285\" (UID: \"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285\") "
Oct 30 00:48:12.120881 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:12.120838    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-maas-api-openshift-ca\") pod \"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285\" (UID: \"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285\") "
Oct 30 00:48:12.120936 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:12.120923    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-kube-api-access-dhsxf\" (UniqueName: \"kubernetes.io/projected/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-kube-api-access-dhsxf\") pod \"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285\" (UID: \"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285\") "
Oct 30 00:48:12.121203 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:12.121173    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-maas-api-openshift-ca" pod "c5d1b67c-e24b-4b75-bf55-c2c7c7d57285" (UID: "c5d1b67c-e24b-4b75-bf55-c2c7c7d57285"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:48:12.121355 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:12.121262    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:12.122824 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:12.122797    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-maas-api-xks-ca" pod "c5d1b67c-e24b-4b75-bf55-c2c7c7d57285" (UID: "c5d1b67c-e24b-4b75-bf55-c2c7c7d57285"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:12.122924 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:12.122881    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-kube-api-access-dhsxf" pod "c5d1b67c-e24b-4b75-bf55-c2c7c7d57285" (UID: "c5d1b67c-e24b-4b75-bf55-c2c7c7d57285"). InnerVolumeSpecName "kube-api-access-dhsxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:12.222265 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:12.222192    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:12.222265 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:12.222220    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhsxf\" (UniqueName: \"kubernetes.io/projected/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285-kube-api-access-dhsxf\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:12.899647 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:12.899617    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs"
Oct 30 00:48:12.900159 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:12.899642    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs" event={"ID":"c5d1b67c-e24b-4b75-bf55-c2c7c7d57285","Type":"ContainerDied","Data":"e3a8942a9ed32d4fd8c45c0287a829080a96e2da0ff7099d144feb5bb332f0db"}
Oct 30 00:48:12.900159 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:12.899686    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3a8942a9ed32d4fd8c45c0287a829080a96e2da0ff7099d144feb5bb332f0db"
Oct 30 00:48:15.920668 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:15.920636    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:48:16.630774 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:48:16.630722    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:48:18.265045 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.265006    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"]
Oct 30 00:48:18.265574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.265556    6099 cpu_manager.go:414] "containerMap: removing container" podUID="c5d1b67c-e24b-4b75-bf55-c2c7c7d57285" containerName="revoke-keys"
Oct 30 00:48:18.265621 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.265578    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d1b67c-e24b-4b75-bf55-c2c7c7d57285" containerName="revoke-keys"
Oct 30 00:48:18.265621 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.265605    6099 cpu_manager.go:414] "containerMap: removing container" podUID="51921f0c-2f68-4220-ba0d-c95ae82ad03f" containerName="revoke-keys"
Oct 30 00:48:18.265621 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.265613    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51921f0c-2f68-4220-ba0d-c95ae82ad03f" containerName="revoke-keys"
Oct 30 00:48:18.265776 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.265764    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d1b67c-e24b-4b75-bf55-c2c7c7d57285" containerName="revoke-keys"
Oct 30 00:48:18.265822 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.265780    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="51921f0c-2f68-4220-ba0d-c95ae82ad03f" containerName="revoke-keys"
Oct 30 00:48:18.269416 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.269397    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.293238 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.293204    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"]
Oct 30 00:48:18.430617 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.430571    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.430617 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.430619    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.430895 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.430721    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.430895 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.430761    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.430895 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.430795    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.430895 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.430869    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czzpj\" (UniqueName: \"kubernetes.io/projected/44d70b91-1209-43ad-b90b-28cdb782b498-kube-api-access-czzpj\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.430895 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.430887    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d70b91-1209-43ad-b90b-28cdb782b498-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.431149 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.431006    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d70b91-1209-43ad-b90b-28cdb782b498-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.431149 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.431099    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d70b91-1209-43ad-b90b-28cdb782b498-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.532292 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.532200    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.532292 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.532261    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.532509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.532296    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.532509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.532322    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.532509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.532369    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.532509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.532429    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czzpj\" (UniqueName: \"kubernetes.io/projected/44d70b91-1209-43ad-b90b-28cdb782b498-kube-api-access-czzpj\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.532509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.532457    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d70b91-1209-43ad-b90b-28cdb782b498-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.532509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.532506    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d70b91-1209-43ad-b90b-28cdb782b498-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.532803 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.532613    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d70b91-1209-43ad-b90b-28cdb782b498-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.532803 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.532748    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.532921 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.532797    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.533026 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.533005    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.533105 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.533035    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.533471 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.533424    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d70b91-1209-43ad-b90b-28cdb782b498-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.535070 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.535046    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.538212 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.538184    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d70b91-1209-43ad-b90b-28cdb782b498-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.556413 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.556385    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d70b91-1209-43ad-b90b-28cdb782b498-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.556699 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.556668    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czzpj\" (UniqueName: \"kubernetes.io/projected/44d70b91-1209-43ad-b90b-28cdb782b498-kube-api-access-czzpj\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.584729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.584693    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:18.901681 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.901646    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"]
Oct 30 00:48:18.904381 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:48:18.904348    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d70b91_1209_43ad_b90b_28cdb782b498.slice/crio-291b072f4e5df52e257667dc7ab251a429f3a20cec61a6590184bae3eb7bc507 WatchSource:0}: Error finding container 291b072f4e5df52e257667dc7ab251a429f3a20cec61a6590184bae3eb7bc507: Status 404 returned error can't find the container with id 291b072f4e5df52e257667dc7ab251a429f3a20cec61a6590184bae3eb7bc507
Oct 30 00:48:18.906650 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.906619    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:48:18.906744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.906687    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:48:18.906788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.906748    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:48:18.948262 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.948239    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs" event={"ID":"44d70b91-1209-43ad-b90b-28cdb782b498","Type":"ContainerStarted","Data":"291b072f4e5df52e257667dc7ab251a429f3a20cec61a6590184bae3eb7bc507"}
Oct 30 00:48:18.990573 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.990535    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds"]
Oct 30 00:48:18.994558 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:18.994536    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds"
Oct 30 00:48:19.007133 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:19.007107    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:48:19.007288 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:19.007216    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:48:19.007288 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:19.007255    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:48:19.007473 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:19.007326    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:48:19.022410 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:19.022379    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds"]
Oct 30 00:48:19.139610 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:19.139574    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a3f843a-ea97-4bb7-8a43-05a44b264942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27aaa04e-vlwds\" (UID: \"5a3f843a-ea97-4bb7-8a43-05a44b264942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds"
Oct 30 00:48:19.139846 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:19.139818    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a3f843a-ea97-4bb7-8a43-05a44b264942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27aaa04e-vlwds\" (UID: \"5a3f843a-ea97-4bb7-8a43-05a44b264942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds"
Oct 30 00:48:19.139981 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:19.139957    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p78nz\" (UniqueName: \"kubernetes.io/projected/5a3f843a-ea97-4bb7-8a43-05a44b264942-kube-api-access-p78nz\") pod \"maas-api-revoke-sub-27aaa04e-vlwds\" (UID: \"5a3f843a-ea97-4bb7-8a43-05a44b264942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds"
Oct 30 00:48:19.240776 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:19.240676    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a3f843a-ea97-4bb7-8a43-05a44b264942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27aaa04e-vlwds\" (UID: \"5a3f843a-ea97-4bb7-8a43-05a44b264942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds"
Oct 30 00:48:19.240940 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:19.240787    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p78nz\" (UniqueName: \"kubernetes.io/projected/5a3f843a-ea97-4bb7-8a43-05a44b264942-kube-api-access-p78nz\") pod \"maas-api-revoke-sub-27aaa04e-vlwds\" (UID: \"5a3f843a-ea97-4bb7-8a43-05a44b264942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds"
Oct 30 00:48:19.240940 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:19.240840    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a3f843a-ea97-4bb7-8a43-05a44b264942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27aaa04e-vlwds\" (UID: \"5a3f843a-ea97-4bb7-8a43-05a44b264942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds"
Oct 30 00:48:19.241471 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:19.241446    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a3f843a-ea97-4bb7-8a43-05a44b264942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27aaa04e-vlwds\" (UID: \"5a3f843a-ea97-4bb7-8a43-05a44b264942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds"
Oct 30 00:48:19.243385 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:19.243360    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a3f843a-ea97-4bb7-8a43-05a44b264942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27aaa04e-vlwds\" (UID: \"5a3f843a-ea97-4bb7-8a43-05a44b264942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds"
Oct 30 00:48:19.262410 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:19.262379    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p78nz\" (UniqueName: \"kubernetes.io/projected/5a3f843a-ea97-4bb7-8a43-05a44b264942-kube-api-access-p78nz\") pod \"maas-api-revoke-sub-27aaa04e-vlwds\" (UID: \"5a3f843a-ea97-4bb7-8a43-05a44b264942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds"
Oct 30 00:48:19.310448 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:19.310412    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds"
Oct 30 00:48:19.629961 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:19.629917    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds"]
Oct 30 00:48:19.632969 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:48:19.632942    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3f843a_ea97_4bb7_8a43_05a44b264942.slice/crio-6544ab76f7e21cbf0852592059f07925bac48d1ca56a4215425a82c5a8fbf15a WatchSource:0}: Error finding container 6544ab76f7e21cbf0852592059f07925bac48d1ca56a4215425a82c5a8fbf15a: Status 404 returned error can't find the container with id 6544ab76f7e21cbf0852592059f07925bac48d1ca56a4215425a82c5a8fbf15a
Oct 30 00:48:19.956936 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:19.956861    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs" event={"ID":"44d70b91-1209-43ad-b90b-28cdb782b498","Type":"ContainerStarted","Data":"8635c17ad8601e737501139dbffd06f53e73a2ea7c871854580bfe31266c14cb"}
Oct 30 00:48:19.958292 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:19.958266    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds" event={"ID":"5a3f843a-ea97-4bb7-8a43-05a44b264942","Type":"ContainerStarted","Data":"6544ab76f7e21cbf0852592059f07925bac48d1ca56a4215425a82c5a8fbf15a"}
Oct 30 00:48:20.001558 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:20.001501    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs" podStartSLOduration=2.001486721 podStartE2EDuration="2.001486721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:19.997285744 +0000 UTC m=+793.854193500" watchObservedRunningTime="2026-10-30 00:48:20.001486721 +0000 UTC m=+793.858394476"
Oct 30 00:48:20.584915 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:20.584880    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:20.589834 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:20.589804    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:48:20.589950 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:20.589915    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs" podUID="44d70b91-1209-43ad-b90b-28cdb782b498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 00:48:20.999716 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:20.999619    6099 generic.go:356] "Generic (PLEG): container finished" podID="5a3f843a-ea97-4bb7-8a43-05a44b264942" containerID="dbec25116c6e6a37627e272a77dbf22b4cbb99dcb4cc25f30abbe2ba6bf1a283" exitCode=0
Oct 30 00:48:20.999858 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:20.999701    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds" event={"ID":"5a3f843a-ea97-4bb7-8a43-05a44b264942","Type":"ContainerDied","Data":"dbec25116c6e6a37627e272a77dbf22b4cbb99dcb4cc25f30abbe2ba6bf1a283"}
Oct 30 00:48:21.589751 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:21.586119    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:48:21.589751 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:21.586181    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs" podUID="44d70b91-1209-43ad-b90b-28cdb782b498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 00:48:22.235344 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:22.235316    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds"
Oct 30 00:48:22.401378 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:22.401230    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a3f843a-ea97-4bb7-8a43-05a44b264942-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a3f843a-ea97-4bb7-8a43-05a44b264942-maas-api-xks-ca\") pod \"5a3f843a-ea97-4bb7-8a43-05a44b264942\" (UID: \"5a3f843a-ea97-4bb7-8a43-05a44b264942\") "
Oct 30 00:48:22.401378 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:22.401300    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3f843a-ea97-4bb7-8a43-05a44b264942-kube-api-access-p78nz\" (UniqueName: \"kubernetes.io/projected/5a3f843a-ea97-4bb7-8a43-05a44b264942-kube-api-access-p78nz\") pod \"5a3f843a-ea97-4bb7-8a43-05a44b264942\" (UID: \"5a3f843a-ea97-4bb7-8a43-05a44b264942\") "
Oct 30 00:48:22.401638 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:22.401561    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a3f843a-ea97-4bb7-8a43-05a44b264942-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a3f843a-ea97-4bb7-8a43-05a44b264942-maas-api-openshift-ca\") pod \"5a3f843a-ea97-4bb7-8a43-05a44b264942\" (UID: \"5a3f843a-ea97-4bb7-8a43-05a44b264942\") "
Oct 30 00:48:22.402078 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:22.402049    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3f843a-ea97-4bb7-8a43-05a44b264942-maas-api-openshift-ca" pod "5a3f843a-ea97-4bb7-8a43-05a44b264942" (UID: "5a3f843a-ea97-4bb7-8a43-05a44b264942"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:48:22.403545 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:22.403509    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3f843a-ea97-4bb7-8a43-05a44b264942-kube-api-access-p78nz" pod "5a3f843a-ea97-4bb7-8a43-05a44b264942" (UID: "5a3f843a-ea97-4bb7-8a43-05a44b264942"). InnerVolumeSpecName "kube-api-access-p78nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:22.404695 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:22.404672    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a3f843a-ea97-4bb7-8a43-05a44b264942-maas-api-xks-ca" pod "5a3f843a-ea97-4bb7-8a43-05a44b264942" (UID: "5a3f843a-ea97-4bb7-8a43-05a44b264942"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:22.503413 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:22.503373    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a3f843a-ea97-4bb7-8a43-05a44b264942-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:22.503413 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:22.503404    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a3f843a-ea97-4bb7-8a43-05a44b264942-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:22.503413 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:22.503414    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p78nz\" (UniqueName: \"kubernetes.io/projected/5a3f843a-ea97-4bb7-8a43-05a44b264942-kube-api-access-p78nz\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:22.589763 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:22.588063    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:48:22.589763 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:22.588124    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs" podUID="44d70b91-1209-43ad-b90b-28cdb782b498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 00:48:23.019185 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.019159    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds"
Oct 30 00:48:23.019185 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.019175    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds" event={"ID":"5a3f843a-ea97-4bb7-8a43-05a44b264942","Type":"ContainerDied","Data":"6544ab76f7e21cbf0852592059f07925bac48d1ca56a4215425a82c5a8fbf15a"}
Oct 30 00:48:23.019393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.019218    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6544ab76f7e21cbf0852592059f07925bac48d1ca56a4215425a82c5a8fbf15a"
Oct 30 00:48:23.387527 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.387489    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs"]
Oct 30 00:48:23.390145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.390110    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5a3f843a-ea97-4bb7-8a43-05a44b264942" containerName="revoke-keys"
Oct 30 00:48:23.390145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.390137    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a3f843a-ea97-4bb7-8a43-05a44b264942" containerName="revoke-keys"
Oct 30 00:48:23.390340 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.390236    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a3f843a-ea97-4bb7-8a43-05a44b264942" containerName="revoke-keys"
Oct 30 00:48:23.393811 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.393793    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs"
Oct 30 00:48:23.406735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.406682    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs"]
Oct 30 00:48:23.414980 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.414100    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:48:23.414980 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.414124    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:48:23.414980 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.414100    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:48:23.414980 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.414147    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:48:23.516619 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.516574    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpjvt\" (UniqueName: \"kubernetes.io/projected/811321cc-7afa-4b87-a18a-b930be1be4c4-kube-api-access-hpjvt\") pod \"maas-api-revoke-sub-ffc215d6-dqqrs\" (UID: \"811321cc-7afa-4b87-a18a-b930be1be4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs"
Oct 30 00:48:23.516619 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.516624    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811321cc-7afa-4b87-a18a-b930be1be4c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffc215d6-dqqrs\" (UID: \"811321cc-7afa-4b87-a18a-b930be1be4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs"
Oct 30 00:48:23.516894 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.516813    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811321cc-7afa-4b87-a18a-b930be1be4c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffc215d6-dqqrs\" (UID: \"811321cc-7afa-4b87-a18a-b930be1be4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs"
Oct 30 00:48:23.589518 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.589088    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:48:23.589518 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.589148    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs" podUID="44d70b91-1209-43ad-b90b-28cdb782b498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 00:48:23.621960 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.620414    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811321cc-7afa-4b87-a18a-b930be1be4c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffc215d6-dqqrs\" (UID: \"811321cc-7afa-4b87-a18a-b930be1be4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs"
Oct 30 00:48:23.621960 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.620639    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpjvt\" (UniqueName: \"kubernetes.io/projected/811321cc-7afa-4b87-a18a-b930be1be4c4-kube-api-access-hpjvt\") pod \"maas-api-revoke-sub-ffc215d6-dqqrs\" (UID: \"811321cc-7afa-4b87-a18a-b930be1be4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs"
Oct 30 00:48:23.621960 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.620672    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811321cc-7afa-4b87-a18a-b930be1be4c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffc215d6-dqqrs\" (UID: \"811321cc-7afa-4b87-a18a-b930be1be4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs"
Oct 30 00:48:23.621960 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.621544    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811321cc-7afa-4b87-a18a-b930be1be4c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffc215d6-dqqrs\" (UID: \"811321cc-7afa-4b87-a18a-b930be1be4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs"
Oct 30 00:48:23.625408 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.625381    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811321cc-7afa-4b87-a18a-b930be1be4c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffc215d6-dqqrs\" (UID: \"811321cc-7afa-4b87-a18a-b930be1be4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs"
Oct 30 00:48:23.653165 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.653107    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpjvt\" (UniqueName: \"kubernetes.io/projected/811321cc-7afa-4b87-a18a-b930be1be4c4-kube-api-access-hpjvt\") pod \"maas-api-revoke-sub-ffc215d6-dqqrs\" (UID: \"811321cc-7afa-4b87-a18a-b930be1be4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs"
Oct 30 00:48:23.707407 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:23.707376    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs"
Oct 30 00:48:24.033792 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:24.033764    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs"]
Oct 30 00:48:24.036137 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:48:24.036105    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811321cc_7afa_4b87_a18a_b930be1be4c4.slice/crio-4124d72d543ff13e5914a4325efe3c564fc6c47f391347f65b76bb88fb89302e WatchSource:0}: Error finding container 4124d72d543ff13e5914a4325efe3c564fc6c47f391347f65b76bb88fb89302e: Status 404 returned error can't find the container with id 4124d72d543ff13e5914a4325efe3c564fc6c47f391347f65b76bb88fb89302e
Oct 30 00:48:24.190729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:24.189108    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj"]
Oct 30 00:48:24.195042 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:24.195021    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj"
Oct 30 00:48:24.203118 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:24.202976    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj"]
Oct 30 00:48:24.329087 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:24.329048    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767c1c10-93bd-4ac3-8b4c-4869626a7b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1286c8aa-6dgtj\" (UID: \"767c1c10-93bd-4ac3-8b4c-4869626a7b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj"
Oct 30 00:48:24.329210 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:24.329141    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-842n9\" (UniqueName: \"kubernetes.io/projected/767c1c10-93bd-4ac3-8b4c-4869626a7b89-kube-api-access-842n9\") pod \"maas-api-revoke-sub-1286c8aa-6dgtj\" (UID: \"767c1c10-93bd-4ac3-8b4c-4869626a7b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj"
Oct 30 00:48:24.329210 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:24.329193    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767c1c10-93bd-4ac3-8b4c-4869626a7b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1286c8aa-6dgtj\" (UID: \"767c1c10-93bd-4ac3-8b4c-4869626a7b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj"
Oct 30 00:48:24.431224 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:24.430271    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767c1c10-93bd-4ac3-8b4c-4869626a7b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1286c8aa-6dgtj\" (UID: \"767c1c10-93bd-4ac3-8b4c-4869626a7b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj"
Oct 30 00:48:24.431224 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:24.430324    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-842n9\" (UniqueName: \"kubernetes.io/projected/767c1c10-93bd-4ac3-8b4c-4869626a7b89-kube-api-access-842n9\") pod \"maas-api-revoke-sub-1286c8aa-6dgtj\" (UID: \"767c1c10-93bd-4ac3-8b4c-4869626a7b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj"
Oct 30 00:48:24.431224 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:24.430392    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767c1c10-93bd-4ac3-8b4c-4869626a7b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1286c8aa-6dgtj\" (UID: \"767c1c10-93bd-4ac3-8b4c-4869626a7b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj"
Oct 30 00:48:24.431224 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:24.431159    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767c1c10-93bd-4ac3-8b4c-4869626a7b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1286c8aa-6dgtj\" (UID: \"767c1c10-93bd-4ac3-8b4c-4869626a7b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj"
Oct 30 00:48:24.433342 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:24.433314    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767c1c10-93bd-4ac3-8b4c-4869626a7b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1286c8aa-6dgtj\" (UID: \"767c1c10-93bd-4ac3-8b4c-4869626a7b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj"
Oct 30 00:48:24.452221 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:24.452189    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-842n9\" (UniqueName: \"kubernetes.io/projected/767c1c10-93bd-4ac3-8b4c-4869626a7b89-kube-api-access-842n9\") pod \"maas-api-revoke-sub-1286c8aa-6dgtj\" (UID: \"767c1c10-93bd-4ac3-8b4c-4869626a7b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj"
Oct 30 00:48:24.512380 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:24.512358    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj"
Oct 30 00:48:24.591870 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:24.590088    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:48:24.591870 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:24.590150    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs" podUID="44d70b91-1209-43ad-b90b-28cdb782b498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 00:48:24.848444 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:24.848414    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj"]
Oct 30 00:48:24.850342 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:48:24.850303    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767c1c10_93bd_4ac3_8b4c_4869626a7b89.slice/crio-a6f1b8a6b0a8fa454fba539d2238ee6905fd122db1a60e81134b931991c1e7ce WatchSource:0}: Error finding container a6f1b8a6b0a8fa454fba539d2238ee6905fd122db1a60e81134b931991c1e7ce: Status 404 returned error can't find the container with id a6f1b8a6b0a8fa454fba539d2238ee6905fd122db1a60e81134b931991c1e7ce
Oct 30 00:48:25.034896 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:25.034852    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs" event={"ID":"811321cc-7afa-4b87-a18a-b930be1be4c4","Type":"ContainerStarted","Data":"044193e9e01b88b1fdc235767efa36bf30de04c4432aacd3bca57c80d24a40f9"}
Oct 30 00:48:25.035066 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:25.034913    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs" event={"ID":"811321cc-7afa-4b87-a18a-b930be1be4c4","Type":"ContainerStarted","Data":"4124d72d543ff13e5914a4325efe3c564fc6c47f391347f65b76bb88fb89302e"}
Oct 30 00:48:25.036312 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:25.036285    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj" event={"ID":"767c1c10-93bd-4ac3-8b4c-4869626a7b89","Type":"ContainerStarted","Data":"a6f1b8a6b0a8fa454fba539d2238ee6905fd122db1a60e81134b931991c1e7ce"}
Oct 30 00:48:25.074741 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:25.073718    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs" podStartSLOduration=2.073688709 podStartE2EDuration="2.073688709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:25.070192289 +0000 UTC m=+798.927100042" watchObservedRunningTime="2026-10-30 00:48:25.073688709 +0000 UTC m=+798.930596465"
Oct 30 00:48:25.769100 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:25.767901    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:25.769100 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:25.768453    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:25.770202 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:25.770174    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:48:26.029356 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:26.029267    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p"]
Oct 30 00:48:26.033569 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:26.033548    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p"
Oct 30 00:48:26.044429 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:26.044408    6099 generic.go:356] "Generic (PLEG): container finished" podID="811321cc-7afa-4b87-a18a-b930be1be4c4" containerID="044193e9e01b88b1fdc235767efa36bf30de04c4432aacd3bca57c80d24a40f9" exitCode=0
Oct 30 00:48:26.044529 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:26.044476    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs" event={"ID":"811321cc-7afa-4b87-a18a-b930be1be4c4","Type":"ContainerDied","Data":"044193e9e01b88b1fdc235767efa36bf30de04c4432aacd3bca57c80d24a40f9"}
Oct 30 00:48:26.047106 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:26.047083    6099 generic.go:356] "Generic (PLEG): container finished" podID="767c1c10-93bd-4ac3-8b4c-4869626a7b89" containerID="e4bca2b37222c641cb18c3aa35c9d631acf406eb28647c9afa91f82c008ed5b3" exitCode=0
Oct 30 00:48:26.047183 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:26.047130    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj" event={"ID":"767c1c10-93bd-4ac3-8b4c-4869626a7b89","Type":"ContainerDied","Data":"e4bca2b37222c641cb18c3aa35c9d631acf406eb28647c9afa91f82c008ed5b3"}
Oct 30 00:48:26.051867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:26.051837    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p"]
Oct 30 00:48:26.057888 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:26.057863    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b661e154-e205-431d-8835-13678bae8175-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3a2d1fa-2hh4p\" (UID: \"b661e154-e205-431d-8835-13678bae8175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p"
Oct 30 00:48:26.058000 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:26.057900    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7mm5\" (UniqueName: \"kubernetes.io/projected/b661e154-e205-431d-8835-13678bae8175-kube-api-access-h7mm5\") pod \"maas-api-revoke-sub-c3a2d1fa-2hh4p\" (UID: \"b661e154-e205-431d-8835-13678bae8175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p"
Oct 30 00:48:26.059099 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:26.058268    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b661e154-e205-431d-8835-13678bae8175-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3a2d1fa-2hh4p\" (UID: \"b661e154-e205-431d-8835-13678bae8175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p"
Oct 30 00:48:26.159423 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:26.159389    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b661e154-e205-431d-8835-13678bae8175-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3a2d1fa-2hh4p\" (UID: \"b661e154-e205-431d-8835-13678bae8175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p"
Oct 30 00:48:26.159589 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:26.159448    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b661e154-e205-431d-8835-13678bae8175-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3a2d1fa-2hh4p\" (UID: \"b661e154-e205-431d-8835-13678bae8175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p"
Oct 30 00:48:26.159589 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:26.159554    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mm5\" (UniqueName: \"kubernetes.io/projected/b661e154-e205-431d-8835-13678bae8175-kube-api-access-h7mm5\") pod \"maas-api-revoke-sub-c3a2d1fa-2hh4p\" (UID: \"b661e154-e205-431d-8835-13678bae8175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p"
Oct 30 00:48:26.160181 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:26.160157    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b661e154-e205-431d-8835-13678bae8175-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3a2d1fa-2hh4p\" (UID: \"b661e154-e205-431d-8835-13678bae8175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p"
Oct 30 00:48:26.161931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:26.161905    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b661e154-e205-431d-8835-13678bae8175-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3a2d1fa-2hh4p\" (UID: \"b661e154-e205-431d-8835-13678bae8175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p"
Oct 30 00:48:26.180701 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:26.180673    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7mm5\" (UniqueName: \"kubernetes.io/projected/b661e154-e205-431d-8835-13678bae8175-kube-api-access-h7mm5\") pod \"maas-api-revoke-sub-c3a2d1fa-2hh4p\" (UID: \"b661e154-e205-431d-8835-13678bae8175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p"
Oct 30 00:48:26.347820 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:26.347793    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p"
Oct 30 00:48:26.643956 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:26.643931    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p"]
Oct 30 00:48:26.646038 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:48:26.646008    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb661e154_e205_431d_8835_13678bae8175.slice/crio-bcff1aa27cbb685f03b39f02e981d371fb8be083949242aa573b495fe90c2ff8 WatchSource:0}: Error finding container bcff1aa27cbb685f03b39f02e981d371fb8be083949242aa573b495fe90c2ff8: Status 404 returned error can't find the container with id bcff1aa27cbb685f03b39f02e981d371fb8be083949242aa573b495fe90c2ff8
Oct 30 00:48:27.054575 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.054541    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p" event={"ID":"b661e154-e205-431d-8835-13678bae8175","Type":"ContainerStarted","Data":"bcff1aa27cbb685f03b39f02e981d371fb8be083949242aa573b495fe90c2ff8"}
Oct 30 00:48:27.209141 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.209113    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs"
Oct 30 00:48:27.236354 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.236330    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj"
Oct 30 00:48:27.271767 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.271732    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/767c1c10-93bd-4ac3-8b4c-4869626a7b89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767c1c10-93bd-4ac3-8b4c-4869626a7b89-maas-api-xks-ca\") pod \"767c1c10-93bd-4ac3-8b4c-4869626a7b89\" (UID: \"767c1c10-93bd-4ac3-8b4c-4869626a7b89\") "
Oct 30 00:48:27.271944 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.271837    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/767c1c10-93bd-4ac3-8b4c-4869626a7b89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767c1c10-93bd-4ac3-8b4c-4869626a7b89-maas-api-openshift-ca\") pod \"767c1c10-93bd-4ac3-8b4c-4869626a7b89\" (UID: \"767c1c10-93bd-4ac3-8b4c-4869626a7b89\") "
Oct 30 00:48:27.271944 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.271894    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811321cc-7afa-4b87-a18a-b930be1be4c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811321cc-7afa-4b87-a18a-b930be1be4c4-maas-api-openshift-ca\") pod \"811321cc-7afa-4b87-a18a-b930be1be4c4\" (UID: \"811321cc-7afa-4b87-a18a-b930be1be4c4\") "
Oct 30 00:48:27.272047 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.271981    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/767c1c10-93bd-4ac3-8b4c-4869626a7b89-kube-api-access-842n9\" (UniqueName: \"kubernetes.io/projected/767c1c10-93bd-4ac3-8b4c-4869626a7b89-kube-api-access-842n9\") pod \"767c1c10-93bd-4ac3-8b4c-4869626a7b89\" (UID: \"767c1c10-93bd-4ac3-8b4c-4869626a7b89\") "
Oct 30 00:48:27.272047 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.272014    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811321cc-7afa-4b87-a18a-b930be1be4c4-kube-api-access-hpjvt\" (UniqueName: \"kubernetes.io/projected/811321cc-7afa-4b87-a18a-b930be1be4c4-kube-api-access-hpjvt\") pod \"811321cc-7afa-4b87-a18a-b930be1be4c4\" (UID: \"811321cc-7afa-4b87-a18a-b930be1be4c4\") "
Oct 30 00:48:27.272167 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.272134    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/811321cc-7afa-4b87-a18a-b930be1be4c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811321cc-7afa-4b87-a18a-b930be1be4c4-maas-api-xks-ca\") pod \"811321cc-7afa-4b87-a18a-b930be1be4c4\" (UID: \"811321cc-7afa-4b87-a18a-b930be1be4c4\") "
Oct 30 00:48:27.272563 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.272538    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811321cc-7afa-4b87-a18a-b930be1be4c4-maas-api-openshift-ca" pod "811321cc-7afa-4b87-a18a-b930be1be4c4" (UID: "811321cc-7afa-4b87-a18a-b930be1be4c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:48:27.273010 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.272992    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767c1c10-93bd-4ac3-8b4c-4869626a7b89-maas-api-openshift-ca" pod "767c1c10-93bd-4ac3-8b4c-4869626a7b89" (UID: "767c1c10-93bd-4ac3-8b4c-4869626a7b89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:48:27.275024 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.274966    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/767c1c10-93bd-4ac3-8b4c-4869626a7b89-maas-api-xks-ca" pod "767c1c10-93bd-4ac3-8b4c-4869626a7b89" (UID: "767c1c10-93bd-4ac3-8b4c-4869626a7b89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:27.275394 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.275364    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811321cc-7afa-4b87-a18a-b930be1be4c4-kube-api-access-hpjvt" pod "811321cc-7afa-4b87-a18a-b930be1be4c4" (UID: "811321cc-7afa-4b87-a18a-b930be1be4c4"). InnerVolumeSpecName "kube-api-access-hpjvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:27.275509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.275480    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767c1c10-93bd-4ac3-8b4c-4869626a7b89-kube-api-access-842n9" pod "767c1c10-93bd-4ac3-8b4c-4869626a7b89" (UID: "767c1c10-93bd-4ac3-8b4c-4869626a7b89"). InnerVolumeSpecName "kube-api-access-842n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:27.277267 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.277242    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/811321cc-7afa-4b87-a18a-b930be1be4c4-maas-api-xks-ca" pod "811321cc-7afa-4b87-a18a-b930be1be4c4" (UID: "811321cc-7afa-4b87-a18a-b930be1be4c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:27.373753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.373678    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-842n9\" (UniqueName: \"kubernetes.io/projected/767c1c10-93bd-4ac3-8b4c-4869626a7b89-kube-api-access-842n9\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:27.373753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.373720    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpjvt\" (UniqueName: \"kubernetes.io/projected/811321cc-7afa-4b87-a18a-b930be1be4c4-kube-api-access-hpjvt\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:27.373753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.373731    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811321cc-7afa-4b87-a18a-b930be1be4c4-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:27.373753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.373740    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767c1c10-93bd-4ac3-8b4c-4869626a7b89-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:27.373753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.373748    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767c1c10-93bd-4ac3-8b4c-4869626a7b89-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:27.373753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:27.373758    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811321cc-7afa-4b87-a18a-b930be1be4c4-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:28.062415 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:28.062331    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs"
Oct 30 00:48:28.062870 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:28.062330    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs" event={"ID":"811321cc-7afa-4b87-a18a-b930be1be4c4","Type":"ContainerDied","Data":"4124d72d543ff13e5914a4325efe3c564fc6c47f391347f65b76bb88fb89302e"}
Oct 30 00:48:28.062870 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:28.062475    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4124d72d543ff13e5914a4325efe3c564fc6c47f391347f65b76bb88fb89302e"
Oct 30 00:48:28.063845 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:28.063827    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj"
Oct 30 00:48:28.063932 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:28.063867    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj" event={"ID":"767c1c10-93bd-4ac3-8b4c-4869626a7b89","Type":"ContainerDied","Data":"a6f1b8a6b0a8fa454fba539d2238ee6905fd122db1a60e81134b931991c1e7ce"}
Oct 30 00:48:28.063932 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:28.063916    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6f1b8a6b0a8fa454fba539d2238ee6905fd122db1a60e81134b931991c1e7ce"
Oct 30 00:48:28.065463 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:28.065435    6099 generic.go:356] "Generic (PLEG): container finished" podID="b661e154-e205-431d-8835-13678bae8175" containerID="f181a1988d8c69e50f32f70ea502a9759c72225447fc1ec54741ee85f9322912" exitCode=0
Oct 30 00:48:28.065569 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:28.065463    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p" event={"ID":"b661e154-e205-431d-8835-13678bae8175","Type":"ContainerDied","Data":"f181a1988d8c69e50f32f70ea502a9759c72225447fc1ec54741ee85f9322912"}
Oct 30 00:48:29.214660 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:29.214632    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p"
Oct 30 00:48:29.295894 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:29.295862    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b661e154-e205-431d-8835-13678bae8175-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b661e154-e205-431d-8835-13678bae8175-maas-api-xks-ca\") pod \"b661e154-e205-431d-8835-13678bae8175\" (UID: \"b661e154-e205-431d-8835-13678bae8175\") "
Oct 30 00:48:29.296071 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:29.295942    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b661e154-e205-431d-8835-13678bae8175-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b661e154-e205-431d-8835-13678bae8175-maas-api-openshift-ca\") pod \"b661e154-e205-431d-8835-13678bae8175\" (UID: \"b661e154-e205-431d-8835-13678bae8175\") "
Oct 30 00:48:29.296071 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:29.296042    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b661e154-e205-431d-8835-13678bae8175-kube-api-access-h7mm5\" (UniqueName: \"kubernetes.io/projected/b661e154-e205-431d-8835-13678bae8175-kube-api-access-h7mm5\") pod \"b661e154-e205-431d-8835-13678bae8175\" (UID: \"b661e154-e205-431d-8835-13678bae8175\") "
Oct 30 00:48:29.297053 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:29.296329    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b661e154-e205-431d-8835-13678bae8175-maas-api-openshift-ca" pod "b661e154-e205-431d-8835-13678bae8175" (UID: "b661e154-e205-431d-8835-13678bae8175"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:48:29.297053 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:29.297010    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b661e154-e205-431d-8835-13678bae8175-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:29.298199 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:29.298177    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b661e154-e205-431d-8835-13678bae8175-kube-api-access-h7mm5" pod "b661e154-e205-431d-8835-13678bae8175" (UID: "b661e154-e205-431d-8835-13678bae8175"). InnerVolumeSpecName "kube-api-access-h7mm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:29.298292 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:29.298203    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b661e154-e205-431d-8835-13678bae8175-maas-api-xks-ca" pod "b661e154-e205-431d-8835-13678bae8175" (UID: "b661e154-e205-431d-8835-13678bae8175"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:29.397987 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:29.397909    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7mm5\" (UniqueName: \"kubernetes.io/projected/b661e154-e205-431d-8835-13678bae8175-kube-api-access-h7mm5\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:29.397987 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:29.397937    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b661e154-e205-431d-8835-13678bae8175-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:30.078149 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:30.078116    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p" event={"ID":"b661e154-e205-431d-8835-13678bae8175","Type":"ContainerDied","Data":"bcff1aa27cbb685f03b39f02e981d371fb8be083949242aa573b495fe90c2ff8"}
Oct 30 00:48:30.078315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:30.078168    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcff1aa27cbb685f03b39f02e981d371fb8be083949242aa573b495fe90c2ff8"
Oct 30 00:48:30.078315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:30.078120    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p"
Oct 30 00:48:38.478373 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.478338    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"]
Oct 30 00:48:38.479144 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.479122    6099 cpu_manager.go:414] "containerMap: removing container" podUID="767c1c10-93bd-4ac3-8b4c-4869626a7b89" containerName="revoke-keys"
Oct 30 00:48:38.479144 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.479144    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767c1c10-93bd-4ac3-8b4c-4869626a7b89" containerName="revoke-keys"
Oct 30 00:48:38.479295 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.479194    6099 cpu_manager.go:414] "containerMap: removing container" podUID="811321cc-7afa-4b87-a18a-b930be1be4c4" containerName="revoke-keys"
Oct 30 00:48:38.479295 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.479204    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811321cc-7afa-4b87-a18a-b930be1be4c4" containerName="revoke-keys"
Oct 30 00:48:38.479295 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.479225    6099 cpu_manager.go:414] "containerMap: removing container" podUID="b661e154-e205-431d-8835-13678bae8175" containerName="revoke-keys"
Oct 30 00:48:38.479295 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.479232    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b661e154-e205-431d-8835-13678bae8175" containerName="revoke-keys"
Oct 30 00:48:38.479414 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.479338    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="767c1c10-93bd-4ac3-8b4c-4869626a7b89" containerName="revoke-keys"
Oct 30 00:48:38.479414 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.479359    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="811321cc-7afa-4b87-a18a-b930be1be4c4" containerName="revoke-keys"
Oct 30 00:48:38.479414 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.479369    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="b661e154-e205-431d-8835-13678bae8175" containerName="revoke-keys"
Oct 30 00:48:38.483549 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.483528    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.500963 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.500940    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"]
Oct 30 00:48:38.508128 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.508105    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6add35\"/\"kube-root-ca.crt\""
Oct 30 00:48:38.508292 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.508237    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6add35\"/\"openshift-service-ca.crt\""
Oct 30 00:48:38.508292 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.508271    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6add35\"/\"default-dockercfg-n8wx7\""
Oct 30 00:48:38.508445 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.508337    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6add35\"/\"distinct-w4-6add35-kserve-self-signed-certs\""
Oct 30 00:48:38.652471 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.652441    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-tmp-dir\") pod \"distinct-w4-6add35-kserve-7c57f95748-gplqq\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.652471 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.652472    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cbwh\" (UniqueName: \"kubernetes.io/projected/69232e0b-1728-4658-874f-e79bbee81329-kube-api-access-8cbwh\") pod \"distinct-w4-6add35-kserve-7c57f95748-gplqq\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.652663 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.652592    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-home\") pod \"distinct-w4-6add35-kserve-7c57f95748-gplqq\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.652663 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.652629    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69232e0b-1728-4658-874f-e79bbee81329-tls-certs\") pod \"distinct-w4-6add35-kserve-7c57f95748-gplqq\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.652779 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.652763    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-model-cache\") pod \"distinct-w4-6add35-kserve-7c57f95748-gplqq\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.652867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.652854    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-dshm\") pod \"distinct-w4-6add35-kserve-7c57f95748-gplqq\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.753848 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.753762    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-dshm\") pod \"distinct-w4-6add35-kserve-7c57f95748-gplqq\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.753848 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.753816    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-tmp-dir\") pod \"distinct-w4-6add35-kserve-7c57f95748-gplqq\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.753848 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.753834    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cbwh\" (UniqueName: \"kubernetes.io/projected/69232e0b-1728-4658-874f-e79bbee81329-kube-api-access-8cbwh\") pod \"distinct-w4-6add35-kserve-7c57f95748-gplqq\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.754141 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.753890    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-home\") pod \"distinct-w4-6add35-kserve-7c57f95748-gplqq\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.754141 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.753915    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69232e0b-1728-4658-874f-e79bbee81329-tls-certs\") pod \"distinct-w4-6add35-kserve-7c57f95748-gplqq\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.754141 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.754013    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-model-cache\") pod \"distinct-w4-6add35-kserve-7c57f95748-gplqq\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.754295 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.754260    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-tmp-dir\") pod \"distinct-w4-6add35-kserve-7c57f95748-gplqq\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.754295 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.754286    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-home\") pod \"distinct-w4-6add35-kserve-7c57f95748-gplqq\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.754471 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.754451    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-model-cache\") pod \"distinct-w4-6add35-kserve-7c57f95748-gplqq\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.756270 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.756250    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-dshm\") pod \"distinct-w4-6add35-kserve-7c57f95748-gplqq\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.756506 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.756489    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69232e0b-1728-4658-874f-e79bbee81329-tls-certs\") pod \"distinct-w4-6add35-kserve-7c57f95748-gplqq\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.774297 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.774272    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cbwh\" (UniqueName: \"kubernetes.io/projected/69232e0b-1728-4658-874f-e79bbee81329-kube-api-access-8cbwh\") pod \"distinct-w4-6add35-kserve-7c57f95748-gplqq\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") " pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:38.797311 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:38.797289    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:39.118272 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:39.118242    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"]
Oct 30 00:48:39.120116 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:48:39.120092    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69232e0b_1728_4658_874f_e79bbee81329.slice/crio-ac1e223c810b85353d22118239f9c460a2775a805418c5ef76d26cea49f1e874 WatchSource:0}: Error finding container ac1e223c810b85353d22118239f9c460a2775a805418c5ef76d26cea49f1e874: Status 404 returned error can't find the container with id ac1e223c810b85353d22118239f9c460a2775a805418c5ef76d26cea49f1e874
Oct 30 00:48:39.163516 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:39.163488    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" event={"ID":"69232e0b-1728-4658-874f-e79bbee81329","Type":"ContainerStarted","Data":"ac1e223c810b85353d22118239f9c460a2775a805418c5ef76d26cea49f1e874"}
Oct 30 00:48:40.171452 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.171411    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" event={"ID":"69232e0b-1728-4658-874f-e79bbee81329","Type":"ContainerStarted","Data":"09714262f7900443f6aaa3d428969b28fc73e84e7522cd292e1ecab94dd25078"}
Oct 30 00:48:40.211034 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.210989    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podStartSLOduration=2.210975892 podStartE2EDuration="2.210975892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:40.20936437 +0000 UTC m=+814.066272138" watchObservedRunningTime="2026-10-30 00:48:40.210975892 +0000 UTC m=+814.067883641"
Oct 30 00:48:40.564472 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.564433    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78"]
Oct 30 00:48:40.568334 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.568310    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78"
Oct 30 00:48:40.581219 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.581195    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:48:40.581329 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.581246    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:48:40.581329 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.581255    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:48:40.590858 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.590839    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:48:40.594031 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.593998    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78"]
Oct 30 00:48:40.677522 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.677481    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtkxf\" (UniqueName: \"kubernetes.io/projected/073ef7a4-c980-4e48-bc70-39903932a68a-kube-api-access-dtkxf\") pod \"maas-api-revoke-sub-a21d539b-6qj78\" (UID: \"073ef7a4-c980-4e48-bc70-39903932a68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78"
Oct 30 00:48:40.677688 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.677629    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073ef7a4-c980-4e48-bc70-39903932a68a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a21d539b-6qj78\" (UID: \"073ef7a4-c980-4e48-bc70-39903932a68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78"
Oct 30 00:48:40.677785 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.677738    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073ef7a4-c980-4e48-bc70-39903932a68a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a21d539b-6qj78\" (UID: \"073ef7a4-c980-4e48-bc70-39903932a68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78"
Oct 30 00:48:40.778939 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.778908    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073ef7a4-c980-4e48-bc70-39903932a68a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a21d539b-6qj78\" (UID: \"073ef7a4-c980-4e48-bc70-39903932a68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78"
Oct 30 00:48:40.779097 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.779017    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073ef7a4-c980-4e48-bc70-39903932a68a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a21d539b-6qj78\" (UID: \"073ef7a4-c980-4e48-bc70-39903932a68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78"
Oct 30 00:48:40.779097 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.779080    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtkxf\" (UniqueName: \"kubernetes.io/projected/073ef7a4-c980-4e48-bc70-39903932a68a-kube-api-access-dtkxf\") pod \"maas-api-revoke-sub-a21d539b-6qj78\" (UID: \"073ef7a4-c980-4e48-bc70-39903932a68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78"
Oct 30 00:48:40.779807 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.779775    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073ef7a4-c980-4e48-bc70-39903932a68a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a21d539b-6qj78\" (UID: \"073ef7a4-c980-4e48-bc70-39903932a68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78"
Oct 30 00:48:40.781608 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.781586    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073ef7a4-c980-4e48-bc70-39903932a68a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a21d539b-6qj78\" (UID: \"073ef7a4-c980-4e48-bc70-39903932a68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78"
Oct 30 00:48:40.797855 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.797833    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:40.802828 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.802807    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtkxf\" (UniqueName: \"kubernetes.io/projected/073ef7a4-c980-4e48-bc70-39903932a68a-kube-api-access-dtkxf\") pod \"maas-api-revoke-sub-a21d539b-6qj78\" (UID: \"073ef7a4-c980-4e48-bc70-39903932a68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78"
Oct 30 00:48:40.882557 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:40.882498    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78"
Oct 30 00:48:41.176877 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:41.176853    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78"]
Oct 30 00:48:42.188192 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:42.188106    6099 generic.go:356] "Generic (PLEG): container finished" podID="073ef7a4-c980-4e48-bc70-39903932a68a" containerID="49c4c00432a29b79b2a0a1a2a143079d1885f40978fd7eeb520d02862cf13d88" exitCode=0
Oct 30 00:48:42.188569 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:42.188187    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78" event={"ID":"073ef7a4-c980-4e48-bc70-39903932a68a","Type":"ContainerDied","Data":"49c4c00432a29b79b2a0a1a2a143079d1885f40978fd7eeb520d02862cf13d88"}
Oct 30 00:48:42.188569 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:42.188248    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78" event={"ID":"073ef7a4-c980-4e48-bc70-39903932a68a","Type":"ContainerStarted","Data":"46a25584f9e5864609050ad4d17eb32b0ca62ffc89b3ff31cc004740acac40b9"}
Oct 30 00:48:43.345894 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:43.345872    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78"
Oct 30 00:48:43.511590 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:43.511508    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/073ef7a4-c980-4e48-bc70-39903932a68a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073ef7a4-c980-4e48-bc70-39903932a68a-maas-api-openshift-ca\") pod \"073ef7a4-c980-4e48-bc70-39903932a68a\" (UID: \"073ef7a4-c980-4e48-bc70-39903932a68a\") "
Oct 30 00:48:43.511751 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:43.511624    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/073ef7a4-c980-4e48-bc70-39903932a68a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073ef7a4-c980-4e48-bc70-39903932a68a-maas-api-xks-ca\") pod \"073ef7a4-c980-4e48-bc70-39903932a68a\" (UID: \"073ef7a4-c980-4e48-bc70-39903932a68a\") "
Oct 30 00:48:43.511751 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:43.511659    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073ef7a4-c980-4e48-bc70-39903932a68a-kube-api-access-dtkxf\" (UniqueName: \"kubernetes.io/projected/073ef7a4-c980-4e48-bc70-39903932a68a-kube-api-access-dtkxf\") pod \"073ef7a4-c980-4e48-bc70-39903932a68a\" (UID: \"073ef7a4-c980-4e48-bc70-39903932a68a\") "
Oct 30 00:48:43.511916 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:43.511889    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073ef7a4-c980-4e48-bc70-39903932a68a-maas-api-openshift-ca" pod "073ef7a4-c980-4e48-bc70-39903932a68a" (UID: "073ef7a4-c980-4e48-bc70-39903932a68a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:48:43.512081 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:43.512060    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073ef7a4-c980-4e48-bc70-39903932a68a-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:43.513671 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:43.513649    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/073ef7a4-c980-4e48-bc70-39903932a68a-maas-api-xks-ca" pod "073ef7a4-c980-4e48-bc70-39903932a68a" (UID: "073ef7a4-c980-4e48-bc70-39903932a68a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:48:43.513748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:43.513694    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073ef7a4-c980-4e48-bc70-39903932a68a-kube-api-access-dtkxf" pod "073ef7a4-c980-4e48-bc70-39903932a68a" (UID: "073ef7a4-c980-4e48-bc70-39903932a68a"). InnerVolumeSpecName "kube-api-access-dtkxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:43.613764 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:43.613735    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073ef7a4-c980-4e48-bc70-39903932a68a-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:43.613764 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:43.613761    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtkxf\" (UniqueName: \"kubernetes.io/projected/073ef7a4-c980-4e48-bc70-39903932a68a-kube-api-access-dtkxf\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:48:44.049558 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:44.049521    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:48:44.201481 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:44.201444    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78" event={"ID":"073ef7a4-c980-4e48-bc70-39903932a68a","Type":"ContainerDied","Data":"46a25584f9e5864609050ad4d17eb32b0ca62ffc89b3ff31cc004740acac40b9"}
Oct 30 00:48:44.201481 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:44.201479    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78"
Oct 30 00:48:44.201684 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:44.201500    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46a25584f9e5864609050ad4d17eb32b0ca62ffc89b3ff31cc004740acac40b9"
Oct 30 00:48:46.663969 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:48:46.663930    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:48:48.797990 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:48.797958    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:48.811816 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:48.811792    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:48.906012 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:48.905970    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"]
Oct 30 00:48:48.906639 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:48.906619    6099 cpu_manager.go:414] "containerMap: removing container" podUID="073ef7a4-c980-4e48-bc70-39903932a68a" containerName="revoke-keys"
Oct 30 00:48:48.906795 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:48.906640    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073ef7a4-c980-4e48-bc70-39903932a68a" containerName="revoke-keys"
Oct 30 00:48:48.906863 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:48.906835    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="073ef7a4-c980-4e48-bc70-39903932a68a" containerName="revoke-keys"
Oct 30 00:48:48.910549 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:48.910530    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:48.928653 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:48.928630    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2b4105-openshift-default-dockercfg-wzpvv\""
Oct 30 00:48:48.931696 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:48.931674    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"]
Oct 30 00:48:49.055301 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.055212    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d5ec0f-0a36-486c-b391-cc0660afa52f-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.055455 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.055326    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.055455 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.055374    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.055455 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.055396    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.055455 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.055417    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.055606 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.055525    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.055606 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.055565    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.055687 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.055618    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w6h5\" (UniqueName: \"kubernetes.io/projected/58d5ec0f-0a36-486c-b391-cc0660afa52f-kube-api-access-9w6h5\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.055810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.055767    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.156833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.156796    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d5ec0f-0a36-486c-b391-cc0660afa52f-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.157036 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.156890    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.157036 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.156947    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.157036 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.156976    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.157036 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.157002    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.157264 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.157061    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.157264 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.157122    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.157264 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.157197    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w6h5\" (UniqueName: \"kubernetes.io/projected/58d5ec0f-0a36-486c-b391-cc0660afa52f-kube-api-access-9w6h5\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.157416 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.157285    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.157568 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.157542    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d5ec0f-0a36-486c-b391-cc0660afa52f-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.157655 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.157634    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.157768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.157741    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.157844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.157821    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.158032 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.157998    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.159535 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.159510    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.160136 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.160119    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.178133 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.178103    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.178286 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.178163    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w6h5\" (UniqueName: \"kubernetes.io/projected/58d5ec0f-0a36-486c-b391-cc0660afa52f-kube-api-access-9w6h5\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.223970 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.223941    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:49.325319 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.325290    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:48:49.565370 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.565346    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"]
Oct 30 00:48:49.567262 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:48:49.567235    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d5ec0f_0a36_486c_b391_cc0660afa52f.slice/crio-ec8f53a910db4b4c32b5313db416065eaade8ebeb6cb49022cbf1fbc29852157 WatchSource:0}: Error finding container ec8f53a910db4b4c32b5313db416065eaade8ebeb6cb49022cbf1fbc29852157: Status 404 returned error can't find the container with id ec8f53a910db4b4c32b5313db416065eaade8ebeb6cb49022cbf1fbc29852157
Oct 30 00:48:49.569463 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.569429    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:48:49.569531 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.569491    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:48:49.569531 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:49.569519    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:48:50.318242 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:50.318199    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh" event={"ID":"58d5ec0f-0a36-486c-b391-cc0660afa52f","Type":"ContainerStarted","Data":"111f83b50a39d29877b14484e33ad579901c11b3bf2cdcfd3e7428db1d3c89bd"}
Oct 30 00:48:50.318733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:50.318264    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh" event={"ID":"58d5ec0f-0a36-486c-b391-cc0660afa52f","Type":"ContainerStarted","Data":"ec8f53a910db4b4c32b5313db416065eaade8ebeb6cb49022cbf1fbc29852157"}
Oct 30 00:48:50.394798 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:50.389076    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh" podStartSLOduration=2.389056323 podStartE2EDuration="2.389056323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:50.386556607 +0000 UTC m=+824.243464366" watchObservedRunningTime="2026-10-30 00:48:50.389056323 +0000 UTC m=+824.245964081"
Oct 30 00:48:51.225939 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.225905    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:51.227461 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.227435    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 00:48:51.227602 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.227482    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh" podUID="58d5ec0f-0a36-486c-b391-cc0660afa52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 00:48:51.349139 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.349102    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"]
Oct 30 00:48:51.354238 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.354210    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.402474 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.402421    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"]
Oct 30 00:48:51.408742 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.408716    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.409831 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.408594    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"]
Oct 30 00:48:51.420091 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.420070    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"]
Oct 30 00:48:51.524210 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.524108    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndzcl\" (UniqueName: \"kubernetes.io/projected/e351b3f4-02ba-48f9-8298-77e6aa2d148e-kube-api-access-ndzcl\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.524210 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.524164    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.524470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.524279    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744aef66-2b84-4c2b-951e-97d85260e902-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.524470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.524396    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.524655 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.524495    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.524655 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.524524    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.524655 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.524547    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc99j\" (UniqueName: \"kubernetes.io/projected/744aef66-2b84-4c2b-951e-97d85260e902-kube-api-access-hc99j\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.524655 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.524592    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.524894 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.524737    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.524894 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.524799    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.524894 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.524831    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.524894 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.524886    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744aef66-2b84-4c2b-951e-97d85260e902-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.525127 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.524922    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.525127 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.524945    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.525127 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.524979    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/744aef66-2b84-4c2b-951e-97d85260e902-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.525127 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.525009    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.525127 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.525049    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.525365 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.525222    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.626305 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.626267    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.626504 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.626329    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.626504 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.626360    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.626504 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.626385    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc99j\" (UniqueName: \"kubernetes.io/projected/744aef66-2b84-4c2b-951e-97d85260e902-kube-api-access-hc99j\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.626695 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.626571    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.626782 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.626701    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.626782 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.626765    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.626782 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.626772    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.626956 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.626798    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.626956 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.626858    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744aef66-2b84-4c2b-951e-97d85260e902-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.626956 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.626893    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.626956 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.626915    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.626956 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.626948    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/744aef66-2b84-4c2b-951e-97d85260e902-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.627199 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.626984    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.627199 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.627026    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.627199 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.627086    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.627199 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.627115    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.627199 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.627160    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.627445 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.627223    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndzcl\" (UniqueName: \"kubernetes.io/projected/e351b3f4-02ba-48f9-8298-77e6aa2d148e-kube-api-access-ndzcl\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.627445 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.627262    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.627445 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.627297    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744aef66-2b84-4c2b-951e-97d85260e902-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.627600 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.627523    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.627664 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.627643    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.627737 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.627676    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.628154 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.628125    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.628154 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.628142    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744aef66-2b84-4c2b-951e-97d85260e902-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.628291 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.628160    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.628350 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.628321    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.630021 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.629993    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744aef66-2b84-4c2b-951e-97d85260e902-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.630131 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.630028    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.630131 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.630060    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.630528 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.630509    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.649248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.649217    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/744aef66-2b84-4c2b-951e-97d85260e902-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.650329 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.650308    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.650730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.650682    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc99j\" (UniqueName: \"kubernetes.io/projected/744aef66-2b84-4c2b-951e-97d85260e902-kube-api-access-hc99j\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:51.650730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.650693    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndzcl\" (UniqueName: \"kubernetes.io/projected/e351b3f4-02ba-48f9-8298-77e6aa2d148e-kube-api-access-ndzcl\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.690199 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.689951    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:51.728425 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:51.728390    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:52.020894 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:52.020868    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"]
Oct 30 00:48:52.022286 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:48:52.022260    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode351b3f4_02ba_48f9_8298_77e6aa2d148e.slice/crio-8d91dc400a95fc9cc76e8d7bb660f19980eb089beb9efa46ad20beee150ca581 WatchSource:0}: Error finding container 8d91dc400a95fc9cc76e8d7bb660f19980eb089beb9efa46ad20beee150ca581: Status 404 returned error can't find the container with id 8d91dc400a95fc9cc76e8d7bb660f19980eb089beb9efa46ad20beee150ca581
Oct 30 00:48:52.024889 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:52.024855    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:48:52.024969 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:52.024918    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:48:52.024969 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:52.024955    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:48:52.097965 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:52.097938    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"]
Oct 30 00:48:52.098604 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:48:52.098581    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744aef66_2b84_4c2b_951e_97d85260e902.slice/crio-fc22126a6e8390e9682d7ceb5634d767054994b5480fa73e78b30ae804240909 WatchSource:0}: Error finding container fc22126a6e8390e9682d7ceb5634d767054994b5480fa73e78b30ae804240909: Status 404 returned error can't find the container with id fc22126a6e8390e9682d7ceb5634d767054994b5480fa73e78b30ae804240909
Oct 30 00:48:52.101087 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:52.101048    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:48:52.101163 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:52.101117    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:48:52.101163 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:52.101150    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:48:52.225302 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:52.224985    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 00:48:52.225302 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:52.225029    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh" podUID="58d5ec0f-0a36-486c-b391-cc0660afa52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 00:48:52.334785 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:52.334744    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h" event={"ID":"e351b3f4-02ba-48f9-8298-77e6aa2d148e","Type":"ContainerStarted","Data":"b4e2bf26a0781bb5e18efa499e132694bcc98d15025c92cb14c54455f6f08731"}
Oct 30 00:48:52.334967 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:52.334809    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h" event={"ID":"e351b3f4-02ba-48f9-8298-77e6aa2d148e","Type":"ContainerStarted","Data":"8d91dc400a95fc9cc76e8d7bb660f19980eb089beb9efa46ad20beee150ca581"}
Oct 30 00:48:52.336652 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:52.336626    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj" event={"ID":"744aef66-2b84-4c2b-951e-97d85260e902","Type":"ContainerStarted","Data":"52151f66ae44ff27f152af16b296853620b4c65214e5e2d4058d3a6a52a765ae"}
Oct 30 00:48:52.336801 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:52.336666    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj" event={"ID":"744aef66-2b84-4c2b-951e-97d85260e902","Type":"ContainerStarted","Data":"fc22126a6e8390e9682d7ceb5634d767054994b5480fa73e78b30ae804240909"}
Oct 30 00:48:52.397784 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:52.397333    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h" podStartSLOduration=1.397311145 podStartE2EDuration="1.397311145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:52.384040658 +0000 UTC m=+826.240948414" watchObservedRunningTime="2026-10-30 00:48:52.397311145 +0000 UTC m=+826.254218901"
Oct 30 00:48:52.425989 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:52.425932    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj" podStartSLOduration=1.425917625 podStartE2EDuration="1.425917625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:52.424443764 +0000 UTC m=+826.281351520" watchObservedRunningTime="2026-10-30 00:48:52.425917625 +0000 UTC m=+826.282825390"
Oct 30 00:48:52.704789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:52.694807    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:48:52.731062 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:52.728809    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:48:53.227730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:53.225231    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 00:48:53.227730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:53.225294    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh" podUID="58d5ec0f-0a36-486c-b391-cc0660afa52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 00:48:53.695728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:53.695576    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:48:53.695728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:53.695635    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h" podUID="e351b3f4-02ba-48f9-8298-77e6aa2d148e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 00:48:53.741341 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:53.740995    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:48:53.741341 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:53.741072    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj" podUID="744aef66-2b84-4c2b-951e-97d85260e902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 00:48:54.231729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:54.231160    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 00:48:54.231729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:54.231239    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh" podUID="58d5ec0f-0a36-486c-b391-cc0660afa52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 00:48:54.692732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:54.692407    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:48:54.692732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:54.692466    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h" podUID="e351b3f4-02ba-48f9-8298-77e6aa2d148e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 00:48:54.729728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:54.729282    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:48:54.729728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:54.729350    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj" podUID="744aef66-2b84-4c2b-951e-97d85260e902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 00:48:55.228178 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:55.228139    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 00:48:55.229607 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:55.229573    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh" podUID="58d5ec0f-0a36-486c-b391-cc0660afa52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 00:48:55.695086 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:55.695051    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:48:55.695394 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:55.695362    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h" podUID="e351b3f4-02ba-48f9-8298-77e6aa2d148e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 00:48:55.728853 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:55.728825    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:48:55.729942 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:55.729908    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj" podUID="744aef66-2b84-4c2b-951e-97d85260e902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 00:48:56.225239 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:56.225208    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 00:48:56.225492 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:56.225471    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh" podUID="58d5ec0f-0a36-486c-b391-cc0660afa52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 00:48:56.693110 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:56.693061    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:48:56.693395 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:56.693365    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h" podUID="e351b3f4-02ba-48f9-8298-77e6aa2d148e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 00:48:56.729514 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:56.729486    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:48:56.729807 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:56.729772    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj" podUID="744aef66-2b84-4c2b-951e-97d85260e902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 00:48:57.229801 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:57.229766    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:57.230267 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:57.230096    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:57.230782 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:57.230762    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 00:48:57.693731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:57.690831    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:48:57.693731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:57.690889    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h" podUID="e351b3f4-02ba-48f9-8298-77e6aa2d148e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 00:48:57.729946 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:57.729912    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:48:57.730218 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:57.730198    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj" podUID="744aef66-2b84-4c2b-951e-97d85260e902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 00:48:58.690906 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:58.690869    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:48:58.691352 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:58.690932    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h" podUID="e351b3f4-02ba-48f9-8298-77e6aa2d148e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 00:48:58.728894 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:58.728860    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:48:58.729038 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:58.728913    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj" podUID="744aef66-2b84-4c2b-951e-97d85260e902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 00:48:59.690904 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:59.690870    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:48:59.691187 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:59.691152    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h" podUID="e351b3f4-02ba-48f9-8298-77e6aa2d148e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 00:48:59.730851 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:59.730814    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:48:59.731038 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:48:59.730888    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj" podUID="744aef66-2b84-4c2b-951e-97d85260e902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 00:49:00.863731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:00.858368    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:49:00.863731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:00.859686    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:49:00.863731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:00.860351    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 00:49:00.974884 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:00.974848    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:49:00.975391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:00.975366    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:49:00.976266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:00.976249    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 00:49:03.521034 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.521002    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"]
Oct 30 00:49:03.526281 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.526261    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.554843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.554816    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"]
Oct 30 00:49:03.610215 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.610175    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.612118 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.612089    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.612262 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.612151    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.612325 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.612272    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.612325 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.612313    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.612428 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.612337    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5edebd8-443c-42d5-afc8-2c5495b4af01-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.612428 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.612378    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.612638 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.612616    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.612773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.612759    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7km9z\" (UniqueName: \"kubernetes.io/projected/e5edebd8-443c-42d5-afc8-2c5495b4af01-kube-api-access-7km9z\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.656016 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.655981    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"]
Oct 30 00:49:03.660172 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.660152    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.683505 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.683474    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"]
Oct 30 00:49:03.714114 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.714082    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.714277 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.714178    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.714277 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.714251    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7km9z\" (UniqueName: \"kubernetes.io/projected/e5edebd8-443c-42d5-afc8-2c5495b4af01-kube-api-access-7km9z\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.714383 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.714286    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.714383 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.714305    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.714383 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.714349    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.714383 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.714379    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.714618 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.714549    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.714618 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.714578    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.714618 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.714620    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5edebd8-443c-42d5-afc8-2c5495b4af01-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.714882 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.714665    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.714882 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.714698    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.714990 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.714874    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.715271 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.715248    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5edebd8-443c-42d5-afc8-2c5495b4af01-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.716697 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.716678    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.716850 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.716833    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.734578 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.734556    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.734748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.734731    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7km9z\" (UniqueName: \"kubernetes.io/projected/e5edebd8-443c-42d5-afc8-2c5495b4af01-kube-api-access-7km9z\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.816027 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.815997    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.816156 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.816037    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c6b739-42b1-4c75-8861-c051595f6620-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.816156 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.816148    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.816245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.816205    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c6b739-42b1-4c75-8861-c051595f6620-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.816245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.816225    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.816338 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.816249    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c6b739-42b1-4c75-8861-c051595f6620-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.816431 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.816406    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pshx\" (UniqueName: \"kubernetes.io/projected/14c6b739-42b1-4c75-8861-c051595f6620-kube-api-access-8pshx\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.816548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.816531    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.816628 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.816563    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.839728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.839684    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:03.917857 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.917827    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c6b739-42b1-4c75-8861-c051595f6620-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.918002 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.917868    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.918002 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.917895    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c6b739-42b1-4c75-8861-c051595f6620-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.918002 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.917966    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pshx\" (UniqueName: \"kubernetes.io/projected/14c6b739-42b1-4c75-8861-c051595f6620-kube-api-access-8pshx\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.918182 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.918066    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.918182 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.918086    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.918182 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.918133    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.918182 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.918160    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c6b739-42b1-4c75-8861-c051595f6620-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.918386 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.918215    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.919765 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.919425    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.919765 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.919688    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.920168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.920025    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.920905 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.920854    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.920994 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.920919    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c6b739-42b1-4c75-8861-c051595f6620-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.921951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.921908    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.922962 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.922937    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c6b739-42b1-4c75-8861-c051595f6620-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.941462 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.941435    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pshx\" (UniqueName: \"kubernetes.io/projected/14c6b739-42b1-4c75-8861-c051595f6620-kube-api-access-8pshx\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.941634 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.941614    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c6b739-42b1-4c75-8861-c051595f6620-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:03.976488 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:03.976466    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:04.159269 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:04.159242    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"]
Oct 30 00:49:04.161463 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:49:04.161431    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5edebd8_443c_42d5_afc8_2c5495b4af01.slice/crio-c6ee597299d9035bac9a52fea0b3dcf10160be2e1760b4fa389e75555094b6e7 WatchSource:0}: Error finding container c6ee597299d9035bac9a52fea0b3dcf10160be2e1760b4fa389e75555094b6e7: Status 404 returned error can't find the container with id c6ee597299d9035bac9a52fea0b3dcf10160be2e1760b4fa389e75555094b6e7
Oct 30 00:49:04.163850 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:04.163814    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:49:04.163926 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:04.163880    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:49:04.163926 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:04.163908    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:49:04.279933 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:04.279905    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"]
Oct 30 00:49:04.281208 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:49:04.281182    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c6b739_42b1_4c75_8861_c051595f6620.slice/crio-458014f2c4348ef76f755bea74250e9a33fa2a68a6cf8f853a39c3d8187e633e WatchSource:0}: Error finding container 458014f2c4348ef76f755bea74250e9a33fa2a68a6cf8f853a39c3d8187e633e: Status 404 returned error can't find the container with id 458014f2c4348ef76f755bea74250e9a33fa2a68a6cf8f853a39c3d8187e633e
Oct 30 00:49:04.284156 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:04.284115    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:49:04.284285 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:04.284201    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:49:04.284285 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:04.284249    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:49:04.493556 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:04.493460    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv" event={"ID":"14c6b739-42b1-4c75-8861-c051595f6620","Type":"ContainerStarted","Data":"a2477eb2294e8e625d92d9f8cb645b085fa4095a83d6e8037856496fad52279e"}
Oct 30 00:49:04.493556 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:04.493525    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv" event={"ID":"14c6b739-42b1-4c75-8861-c051595f6620","Type":"ContainerStarted","Data":"458014f2c4348ef76f755bea74250e9a33fa2a68a6cf8f853a39c3d8187e633e"}
Oct 30 00:49:04.497243 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:04.497207    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp" event={"ID":"e5edebd8-443c-42d5-afc8-2c5495b4af01","Type":"ContainerStarted","Data":"96c5b021e92d1ef9159f2cad45b6085bbd0fe0bc3277f456b8f35733b29fb1d6"}
Oct 30 00:49:04.497386 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:04.497266    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp" event={"ID":"e5edebd8-443c-42d5-afc8-2c5495b4af01","Type":"ContainerStarted","Data":"c6ee597299d9035bac9a52fea0b3dcf10160be2e1760b4fa389e75555094b6e7"}
Oct 30 00:49:04.536291 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:04.536228    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv" podStartSLOduration=1.5362087020000001 podStartE2EDuration="1.536208702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:04.533605138 +0000 UTC m=+838.390512896" watchObservedRunningTime="2026-10-30 00:49:04.536208702 +0000 UTC m=+838.393116458"
Oct 30 00:49:04.577512 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:04.577461    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp" podStartSLOduration=1.577443359 podStartE2EDuration="1.577443359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:04.574148602 +0000 UTC m=+838.431056356" watchObservedRunningTime="2026-10-30 00:49:04.577443359 +0000 UTC m=+838.434351115"
Oct 30 00:49:04.840129 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:04.840095    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:04.978864 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:04.978820    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:05.842097 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:05.842063    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:49:05.842561 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:05.842124    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp" podUID="e5edebd8-443c-42d5-afc8-2c5495b4af01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 00:49:05.982729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:05.980900    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:49:05.982729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:05.980968    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv" podUID="14c6b739-42b1-4c75-8861-c051595f6620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 00:49:06.840213 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:06.840179    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:49:06.840398 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:06.840295    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp" podUID="e5edebd8-443c-42d5-afc8-2c5495b4af01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 00:49:06.983730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:06.979181    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:49:06.983730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:06.979259    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv" podUID="14c6b739-42b1-4c75-8861-c051595f6620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 00:49:07.840100 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:07.840059    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:49:07.840281 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:07.840117    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp" podUID="e5edebd8-443c-42d5-afc8-2c5495b4af01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 00:49:07.993734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:07.993462    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:49:07.993734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:07.993529    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv" podUID="14c6b739-42b1-4c75-8861-c051595f6620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 00:49:08.840284 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:08.840247    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:49:08.840464 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:08.840301    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp" podUID="e5edebd8-443c-42d5-afc8-2c5495b4af01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 00:49:08.980226 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:08.980190    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:49:08.980541 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:08.980495    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv" podUID="14c6b739-42b1-4c75-8861-c051595f6620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 00:49:09.840008 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:09.839974    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:49:09.840461 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:09.840025    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp" podUID="e5edebd8-443c-42d5-afc8-2c5495b4af01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 00:49:09.986728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:09.986019    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:49:09.986728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:09.986084    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv" podUID="14c6b739-42b1-4c75-8861-c051595f6620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 00:49:10.403099 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:10.403056    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:49:10.840203 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:10.840164    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:49:10.840601 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:10.840230    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp" podUID="e5edebd8-443c-42d5-afc8-2c5495b4af01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 00:49:11.847065 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:11.847032    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:11.849066 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:11.849044    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:11.853468 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:11.853449    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 00:49:11.980047 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:11.979988    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:49:11.989276 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:11.989237    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv" podUID="14c6b739-42b1-4c75-8861-c051595f6620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:12.793730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:12.793172    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:12.796428 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:12.794161    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:12.796428 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:12.796368    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:49:16.636324 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:49:16.636257    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:49:22.626957 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:22.626924    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt"]
Oct 30 00:49:22.631098 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:22.631080    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt"
Oct 30 00:49:22.643446 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:22.643426    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:49:22.653618 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:22.653591    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:49:22.653930 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:22.653896    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:49:22.653930 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:22.653917    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:49:22.658013 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:22.657988    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt"]
Oct 30 00:49:22.772016 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:22.771981    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3ac3dc-adee-4991-949c-824aefcb7a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b93bb10-gprtt\" (UID: \"4c3ac3dc-adee-4991-949c-824aefcb7a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt"
Oct 30 00:49:22.772198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:22.772028    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kll6\" (UniqueName: \"kubernetes.io/projected/4c3ac3dc-adee-4991-949c-824aefcb7a28-kube-api-access-8kll6\") pod \"maas-api-revoke-sub-8b93bb10-gprtt\" (UID: \"4c3ac3dc-adee-4991-949c-824aefcb7a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt"
Oct 30 00:49:22.772198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:22.772154    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3ac3dc-adee-4991-949c-824aefcb7a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b93bb10-gprtt\" (UID: \"4c3ac3dc-adee-4991-949c-824aefcb7a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt"
Oct 30 00:49:22.873939 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:22.873771    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3ac3dc-adee-4991-949c-824aefcb7a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b93bb10-gprtt\" (UID: \"4c3ac3dc-adee-4991-949c-824aefcb7a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt"
Oct 30 00:49:22.874122 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:22.874007    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3ac3dc-adee-4991-949c-824aefcb7a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b93bb10-gprtt\" (UID: \"4c3ac3dc-adee-4991-949c-824aefcb7a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt"
Oct 30 00:49:22.874122 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:22.874046    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kll6\" (UniqueName: \"kubernetes.io/projected/4c3ac3dc-adee-4991-949c-824aefcb7a28-kube-api-access-8kll6\") pod \"maas-api-revoke-sub-8b93bb10-gprtt\" (UID: \"4c3ac3dc-adee-4991-949c-824aefcb7a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt"
Oct 30 00:49:22.875055 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:22.875023    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3ac3dc-adee-4991-949c-824aefcb7a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b93bb10-gprtt\" (UID: \"4c3ac3dc-adee-4991-949c-824aefcb7a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt"
Oct 30 00:49:22.876934 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:22.876909    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3ac3dc-adee-4991-949c-824aefcb7a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b93bb10-gprtt\" (UID: \"4c3ac3dc-adee-4991-949c-824aefcb7a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt"
Oct 30 00:49:22.895101 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:22.895036    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kll6\" (UniqueName: \"kubernetes.io/projected/4c3ac3dc-adee-4991-949c-824aefcb7a28-kube-api-access-8kll6\") pod \"maas-api-revoke-sub-8b93bb10-gprtt\" (UID: \"4c3ac3dc-adee-4991-949c-824aefcb7a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt"
Oct 30 00:49:22.944623 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:22.944594    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt"
Oct 30 00:49:23.155154 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:23.155079    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf"]
Oct 30 00:49:23.159452 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:23.159417    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf"
Oct 30 00:49:23.178246 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:23.178213    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf"]
Oct 30 00:49:23.242629 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:23.242590    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt"]
Oct 30 00:49:23.244589 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:49:23.244560    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3ac3dc_adee_4991_949c_824aefcb7a28.slice/crio-3ec1edbb634bde3411e29138df2a2c6b22966b073a3dea1fbb697b5783cce8d9 WatchSource:0}: Error finding container 3ec1edbb634bde3411e29138df2a2c6b22966b073a3dea1fbb697b5783cce8d9: Status 404 returned error can't find the container with id 3ec1edbb634bde3411e29138df2a2c6b22966b073a3dea1fbb697b5783cce8d9
Oct 30 00:49:23.280109 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:23.280081    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2rlf\" (UniqueName: \"kubernetes.io/projected/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-kube-api-access-b2rlf\") pod \"maas-api-revoke-sub-1b8bd8a4-wx6rf\" (UID: \"f7f065b6-13ab-455d-a287-2e2a1fedaa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf"
Oct 30 00:49:23.280210 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:23.280135    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b8bd8a4-wx6rf\" (UID: \"f7f065b6-13ab-455d-a287-2e2a1fedaa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf"
Oct 30 00:49:23.280305 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:23.280281    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b8bd8a4-wx6rf\" (UID: \"f7f065b6-13ab-455d-a287-2e2a1fedaa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf"
Oct 30 00:49:23.381336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:23.381304    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2rlf\" (UniqueName: \"kubernetes.io/projected/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-kube-api-access-b2rlf\") pod \"maas-api-revoke-sub-1b8bd8a4-wx6rf\" (UID: \"f7f065b6-13ab-455d-a287-2e2a1fedaa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf"
Oct 30 00:49:23.381512 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:23.381432    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b8bd8a4-wx6rf\" (UID: \"f7f065b6-13ab-455d-a287-2e2a1fedaa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf"
Oct 30 00:49:23.381581 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:23.381554    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b8bd8a4-wx6rf\" (UID: \"f7f065b6-13ab-455d-a287-2e2a1fedaa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf"
Oct 30 00:49:23.382377 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:23.382346    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b8bd8a4-wx6rf\" (UID: \"f7f065b6-13ab-455d-a287-2e2a1fedaa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf"
Oct 30 00:49:23.384125 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:23.384098    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b8bd8a4-wx6rf\" (UID: \"f7f065b6-13ab-455d-a287-2e2a1fedaa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf"
Oct 30 00:49:23.403811 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:23.403790    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2rlf\" (UniqueName: \"kubernetes.io/projected/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-kube-api-access-b2rlf\") pod \"maas-api-revoke-sub-1b8bd8a4-wx6rf\" (UID: \"f7f065b6-13ab-455d-a287-2e2a1fedaa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf"
Oct 30 00:49:23.474907 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:23.474854    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf"
Oct 30 00:49:23.676224 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:23.676187    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt" event={"ID":"4c3ac3dc-adee-4991-949c-824aefcb7a28","Type":"ContainerStarted","Data":"474a497954e2e8b8124542c734893f04f5f77abbed6daca4ba2756c9e12a9ea4"}
Oct 30 00:49:23.676695 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:23.676281    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt" event={"ID":"4c3ac3dc-adee-4991-949c-824aefcb7a28","Type":"ContainerStarted","Data":"3ec1edbb634bde3411e29138df2a2c6b22966b073a3dea1fbb697b5783cce8d9"}
Oct 30 00:49:23.718808 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:23.718759    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt" podStartSLOduration=1.7187456399999999 podStartE2EDuration="1.71874564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:23.715232476 +0000 UTC m=+857.572140231" watchObservedRunningTime="2026-10-30 00:49:23.71874564 +0000 UTC m=+857.575653394"
Oct 30 00:49:23.855412 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:23.855384    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf"]
Oct 30 00:49:23.855746 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:49:23.855720    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f065b6_13ab_455d_a287_2e2a1fedaa5e.slice/crio-51a52dee1ef2b16c08d729497bf7df1df7a1440dd9f38f24f0046fe6a29ee1ba WatchSource:0}: Error finding container 51a52dee1ef2b16c08d729497bf7df1df7a1440dd9f38f24f0046fe6a29ee1ba: Status 404 returned error can't find the container with id 51a52dee1ef2b16c08d729497bf7df1df7a1440dd9f38f24f0046fe6a29ee1ba
Oct 30 00:49:24.684399 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:24.684357    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf" event={"ID":"f7f065b6-13ab-455d-a287-2e2a1fedaa5e","Type":"ContainerStarted","Data":"9575cd71200b8152996772a30a79107c79a9c2c9c2d5a2513e0c15d3bf049223"}
Oct 30 00:49:24.684902 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:24.684422    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf" event={"ID":"f7f065b6-13ab-455d-a287-2e2a1fedaa5e","Type":"ContainerStarted","Data":"51a52dee1ef2b16c08d729497bf7df1df7a1440dd9f38f24f0046fe6a29ee1ba"}
Oct 30 00:49:24.686299 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:24.686267    6099 generic.go:356] "Generic (PLEG): container finished" podID="4c3ac3dc-adee-4991-949c-824aefcb7a28" containerID="474a497954e2e8b8124542c734893f04f5f77abbed6daca4ba2756c9e12a9ea4" exitCode=0
Oct 30 00:49:24.686414 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:24.686336    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt" event={"ID":"4c3ac3dc-adee-4991-949c-824aefcb7a28","Type":"ContainerDied","Data":"474a497954e2e8b8124542c734893f04f5f77abbed6daca4ba2756c9e12a9ea4"}
Oct 30 00:49:24.722288 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:24.722239    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf" podStartSLOduration=1.7222247899999998 podStartE2EDuration="1.72222479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:24.720666746 +0000 UTC m=+858.577574502" watchObservedRunningTime="2026-10-30 00:49:24.72222479 +0000 UTC m=+858.579132545"
Oct 30 00:49:25.695212 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:25.695172    6099 generic.go:356] "Generic (PLEG): container finished" podID="f7f065b6-13ab-455d-a287-2e2a1fedaa5e" containerID="9575cd71200b8152996772a30a79107c79a9c2c9c2d5a2513e0c15d3bf049223" exitCode=0
Oct 30 00:49:25.695643 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:25.695219    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf" event={"ID":"f7f065b6-13ab-455d-a287-2e2a1fedaa5e","Type":"ContainerDied","Data":"9575cd71200b8152996772a30a79107c79a9c2c9c2d5a2513e0c15d3bf049223"}
Oct 30 00:49:25.873925 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:25.873901    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt"
Oct 30 00:49:25.911875 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:25.911839    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c3ac3dc-adee-4991-949c-824aefcb7a28-kube-api-access-8kll6\" (UniqueName: \"kubernetes.io/projected/4c3ac3dc-adee-4991-949c-824aefcb7a28-kube-api-access-8kll6\") pod \"4c3ac3dc-adee-4991-949c-824aefcb7a28\" (UID: \"4c3ac3dc-adee-4991-949c-824aefcb7a28\") "
Oct 30 00:49:25.912039 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:25.912025    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c3ac3dc-adee-4991-949c-824aefcb7a28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3ac3dc-adee-4991-949c-824aefcb7a28-maas-api-openshift-ca\") pod \"4c3ac3dc-adee-4991-949c-824aefcb7a28\" (UID: \"4c3ac3dc-adee-4991-949c-824aefcb7a28\") "
Oct 30 00:49:25.912219 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:25.912200    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c3ac3dc-adee-4991-949c-824aefcb7a28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3ac3dc-adee-4991-949c-824aefcb7a28-maas-api-xks-ca\") pod \"4c3ac3dc-adee-4991-949c-824aefcb7a28\" (UID: \"4c3ac3dc-adee-4991-949c-824aefcb7a28\") "
Oct 30 00:49:25.912430 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:25.912408    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c3ac3dc-adee-4991-949c-824aefcb7a28-maas-api-openshift-ca" pod "4c3ac3dc-adee-4991-949c-824aefcb7a28" (UID: "4c3ac3dc-adee-4991-949c-824aefcb7a28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:49:25.912787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:25.912766    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3ac3dc-adee-4991-949c-824aefcb7a28-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:25.914544 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:25.914518    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c3ac3dc-adee-4991-949c-824aefcb7a28-kube-api-access-8kll6" pod "4c3ac3dc-adee-4991-949c-824aefcb7a28" (UID: "4c3ac3dc-adee-4991-949c-824aefcb7a28"). InnerVolumeSpecName "kube-api-access-8kll6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:49:25.914653 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:25.914543    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c3ac3dc-adee-4991-949c-824aefcb7a28-maas-api-xks-ca" pod "4c3ac3dc-adee-4991-949c-824aefcb7a28" (UID: "4c3ac3dc-adee-4991-949c-824aefcb7a28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:49:26.014190 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:26.014119    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kll6\" (UniqueName: \"kubernetes.io/projected/4c3ac3dc-adee-4991-949c-824aefcb7a28-kube-api-access-8kll6\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:26.014190 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:26.014145    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3ac3dc-adee-4991-949c-824aefcb7a28-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:26.704636 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:26.704608    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt"
Oct 30 00:49:26.705034 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:26.704648    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt" event={"ID":"4c3ac3dc-adee-4991-949c-824aefcb7a28","Type":"ContainerDied","Data":"3ec1edbb634bde3411e29138df2a2c6b22966b073a3dea1fbb697b5783cce8d9"}
Oct 30 00:49:26.705034 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:26.704701    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ec1edbb634bde3411e29138df2a2c6b22966b073a3dea1fbb697b5783cce8d9"
Oct 30 00:49:26.887930 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:26.887903    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf"
Oct 30 00:49:27.039606 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:27.039577    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-maas-api-xks-ca\") pod \"f7f065b6-13ab-455d-a287-2e2a1fedaa5e\" (UID: \"f7f065b6-13ab-455d-a287-2e2a1fedaa5e\") "
Oct 30 00:49:27.039776 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:27.039654    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-kube-api-access-b2rlf\" (UniqueName: \"kubernetes.io/projected/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-kube-api-access-b2rlf\") pod \"f7f065b6-13ab-455d-a287-2e2a1fedaa5e\" (UID: \"f7f065b6-13ab-455d-a287-2e2a1fedaa5e\") "
Oct 30 00:49:27.039776 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:27.039739    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-maas-api-openshift-ca\") pod \"f7f065b6-13ab-455d-a287-2e2a1fedaa5e\" (UID: \"f7f065b6-13ab-455d-a287-2e2a1fedaa5e\") "
Oct 30 00:49:27.040114 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:27.040086    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-maas-api-openshift-ca" pod "f7f065b6-13ab-455d-a287-2e2a1fedaa5e" (UID: "f7f065b6-13ab-455d-a287-2e2a1fedaa5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:49:27.041772 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:27.041746    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-maas-api-xks-ca" pod "f7f065b6-13ab-455d-a287-2e2a1fedaa5e" (UID: "f7f065b6-13ab-455d-a287-2e2a1fedaa5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:49:27.041869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:27.041838    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-kube-api-access-b2rlf" pod "f7f065b6-13ab-455d-a287-2e2a1fedaa5e" (UID: "f7f065b6-13ab-455d-a287-2e2a1fedaa5e"). InnerVolumeSpecName "kube-api-access-b2rlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:49:27.140806 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:27.140778    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:27.140806 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:27.140806    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:27.140946 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:27.140820    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2rlf\" (UniqueName: \"kubernetes.io/projected/f7f065b6-13ab-455d-a287-2e2a1fedaa5e-kube-api-access-b2rlf\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:27.712533 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:27.712501    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf" event={"ID":"f7f065b6-13ab-455d-a287-2e2a1fedaa5e","Type":"ContainerDied","Data":"51a52dee1ef2b16c08d729497bf7df1df7a1440dd9f38f24f0046fe6a29ee1ba"}
Oct 30 00:49:27.712533 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:27.712519    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf"
Oct 30 00:49:27.712990 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:27.712547    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51a52dee1ef2b16c08d729497bf7df1df7a1440dd9f38f24f0046fe6a29ee1ba"
Oct 30 00:49:28.703543 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:28.703510    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:49:31.662787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.662752    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"]
Oct 30 00:49:31.663662 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.663637    6099 cpu_manager.go:414] "containerMap: removing container" podUID="f7f065b6-13ab-455d-a287-2e2a1fedaa5e" containerName="revoke-keys"
Oct 30 00:49:31.663662 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.663663    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f065b6-13ab-455d-a287-2e2a1fedaa5e" containerName="revoke-keys"
Oct 30 00:49:31.663860 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.663753    6099 cpu_manager.go:414] "containerMap: removing container" podUID="4c3ac3dc-adee-4991-949c-824aefcb7a28" containerName="revoke-keys"
Oct 30 00:49:31.663860 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.663765    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c3ac3dc-adee-4991-949c-824aefcb7a28" containerName="revoke-keys"
Oct 30 00:49:31.663973 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.663922    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c3ac3dc-adee-4991-949c-824aefcb7a28" containerName="revoke-keys"
Oct 30 00:49:31.663973 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.663944    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f065b6-13ab-455d-a287-2e2a1fedaa5e" containerName="revoke-keys"
Oct 30 00:49:31.668401 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.668384    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.691299 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.691276    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-89a676\"/\"distinct-2-w5-89a676-kserve-self-signed-certs\""
Oct 30 00:49:31.695849 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.695824    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"]
Oct 30 00:49:31.713864 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.713835    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3662307-8768-40a7-91f4-1d36cc1b41b9-tls-certs\") pod \"distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.713964 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.713874    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2tmk\" (UniqueName: \"kubernetes.io/projected/c3662307-8768-40a7-91f4-1d36cc1b41b9-kube-api-access-q2tmk\") pod \"distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.713964 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.713903    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-model-cache\") pod \"distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.714043 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.713994    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-tmp-dir\") pod \"distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.714082 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.714064    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-dshm\") pod \"distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.714185 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.714162    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-home\") pod \"distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.815324 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.815293    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3662307-8768-40a7-91f4-1d36cc1b41b9-tls-certs\") pod \"distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.815487 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.815333    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2tmk\" (UniqueName: \"kubernetes.io/projected/c3662307-8768-40a7-91f4-1d36cc1b41b9-kube-api-access-q2tmk\") pod \"distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.815487 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.815352    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-model-cache\") pod \"distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.815487 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.815394    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-tmp-dir\") pod \"distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.815487 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.815475    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-dshm\") pod \"distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.815698 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.815524    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-home\") pod \"distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.815977 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.815945    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-model-cache\") pod \"distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.816122 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.816019    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-tmp-dir\") pod \"distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.816122 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.816052    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-home\") pod \"distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.817806 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.817786    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-dshm\") pod \"distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.817965 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.817948    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3662307-8768-40a7-91f4-1d36cc1b41b9-tls-certs\") pod \"distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.835633 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.835610    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2tmk\" (UniqueName: \"kubernetes.io/projected/c3662307-8768-40a7-91f4-1d36cc1b41b9-kube-api-access-q2tmk\") pod \"distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") " pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:31.982340 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:31.982254    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:32.286463 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:32.286437    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"]
Oct 30 00:49:32.287413 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:49:32.287392    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3662307_8768_40a7_91f4_1d36cc1b41b9.slice/crio-1591b2c398995f1edacd057f14ea7f9f1f973a93c05524a2f413f1836e638442 WatchSource:0}: Error finding container 1591b2c398995f1edacd057f14ea7f9f1f973a93c05524a2f413f1836e638442: Status 404 returned error can't find the container with id 1591b2c398995f1edacd057f14ea7f9f1f973a93c05524a2f413f1836e638442
Oct 30 00:49:32.749843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:32.749799    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" event={"ID":"c3662307-8768-40a7-91f4-1d36cc1b41b9","Type":"ContainerStarted","Data":"92d3ce385dd6fdf638aef991a6ea7819b4cb9edeb9041bce39f47c345778d075"}
Oct 30 00:49:32.750250 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:32.749853    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" event={"ID":"c3662307-8768-40a7-91f4-1d36cc1b41b9","Type":"ContainerStarted","Data":"1591b2c398995f1edacd057f14ea7f9f1f973a93c05524a2f413f1836e638442"}
Oct 30 00:49:32.786476 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:32.786391    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podStartSLOduration=1.786379405 podStartE2EDuration="1.786379405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:32.78604035 +0000 UTC m=+866.642948105" watchObservedRunningTime="2026-10-30 00:49:32.786379405 +0000 UTC m=+866.643287160"
Oct 30 00:49:32.907345 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:32.907314    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw"]
Oct 30 00:49:32.912368 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:32.912275    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw"
Oct 30 00:49:32.927126 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:32.927096    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:49:32.927302 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:32.927159    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:49:32.927423 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:32.927221    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:49:32.927423 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:32.927418    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:49:32.930853 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:32.930830    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw"]
Oct 30 00:49:32.982400 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:32.982367    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:33.033390 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:33.033310    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d85827-e8ab-4ca2-9e90-25dd27204143-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d283929f-ckrdw\" (UID: \"46d85827-e8ab-4ca2-9e90-25dd27204143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw"
Oct 30 00:49:33.033390 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:33.033361    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d85827-e8ab-4ca2-9e90-25dd27204143-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d283929f-ckrdw\" (UID: \"46d85827-e8ab-4ca2-9e90-25dd27204143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw"
Oct 30 00:49:33.033577 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:33.033553    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lxmm\" (UniqueName: \"kubernetes.io/projected/46d85827-e8ab-4ca2-9e90-25dd27204143-kube-api-access-6lxmm\") pod \"maas-api-revoke-sub-d283929f-ckrdw\" (UID: \"46d85827-e8ab-4ca2-9e90-25dd27204143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw"
Oct 30 00:49:33.134817 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:33.134786    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lxmm\" (UniqueName: \"kubernetes.io/projected/46d85827-e8ab-4ca2-9e90-25dd27204143-kube-api-access-6lxmm\") pod \"maas-api-revoke-sub-d283929f-ckrdw\" (UID: \"46d85827-e8ab-4ca2-9e90-25dd27204143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw"
Oct 30 00:49:33.135006 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:33.134899    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d85827-e8ab-4ca2-9e90-25dd27204143-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d283929f-ckrdw\" (UID: \"46d85827-e8ab-4ca2-9e90-25dd27204143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw"
Oct 30 00:49:33.135006 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:33.134925    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d85827-e8ab-4ca2-9e90-25dd27204143-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d283929f-ckrdw\" (UID: \"46d85827-e8ab-4ca2-9e90-25dd27204143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw"
Oct 30 00:49:33.135686 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:33.135664    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d85827-e8ab-4ca2-9e90-25dd27204143-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d283929f-ckrdw\" (UID: \"46d85827-e8ab-4ca2-9e90-25dd27204143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw"
Oct 30 00:49:33.137264 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:33.137240    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d85827-e8ab-4ca2-9e90-25dd27204143-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d283929f-ckrdw\" (UID: \"46d85827-e8ab-4ca2-9e90-25dd27204143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw"
Oct 30 00:49:33.155515 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:33.155493    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lxmm\" (UniqueName: \"kubernetes.io/projected/46d85827-e8ab-4ca2-9e90-25dd27204143-kube-api-access-6lxmm\") pod \"maas-api-revoke-sub-d283929f-ckrdw\" (UID: \"46d85827-e8ab-4ca2-9e90-25dd27204143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw"
Oct 30 00:49:33.229516 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:33.229486    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw"
Oct 30 00:49:33.525103 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:33.525068    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw"]
Oct 30 00:49:33.527195 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:49:33.527163    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d85827_e8ab_4ca2_9e90_25dd27204143.slice/crio-3500a1171f213c332a9ff266d69f325bead2078aa07ca71f69e273592b785590 WatchSource:0}: Error finding container 3500a1171f213c332a9ff266d69f325bead2078aa07ca71f69e273592b785590: Status 404 returned error can't find the container with id 3500a1171f213c332a9ff266d69f325bead2078aa07ca71f69e273592b785590
Oct 30 00:49:33.757926 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:33.757895    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw" event={"ID":"46d85827-e8ab-4ca2-9e90-25dd27204143","Type":"ContainerStarted","Data":"3500a1171f213c332a9ff266d69f325bead2078aa07ca71f69e273592b785590"}
Oct 30 00:49:33.981448 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:33.981415    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"]
Oct 30 00:49:33.985931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:33.985911    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.012366 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.012338    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"]
Oct 30 00:49:34.046195 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.046164    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.046330 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.046214    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.046330 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.046236    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tzd7\" (UniqueName: \"kubernetes.io/projected/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-kube-api-access-7tzd7\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.046330 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.046309    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.046431 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.046351    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.046462 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.046431    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.046496 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.046479    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.046599 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.046575    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.046795 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.046769    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.148314 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.148233    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.148314 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.148275    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.148552 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.148358    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.148552 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.148417    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.148552 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.148455    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.148552 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.148478    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tzd7\" (UniqueName: \"kubernetes.io/projected/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-kube-api-access-7tzd7\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.148552 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.148508    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.148843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.148567    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.148843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.148608    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.148843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.148794    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.149008 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.148968    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.149091 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.149055    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.149155 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.149114    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.149155 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.149130    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.150823 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.150798    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.151065 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.151047    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.169364 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.169329    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tzd7\" (UniqueName: \"kubernetes.io/projected/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-kube-api-access-7tzd7\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.169465 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.169336    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.303430 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.303399    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:34.617155 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.617127    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"]
Oct 30 00:49:34.618140 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:49:34.618116    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c1fcbd_ab57_49f6_a3e5_92eb3a95a999.slice/crio-35d09ad1c61512e0da21d29f17cd9c696801398a17956b5de309369190f381e8 WatchSource:0}: Error finding container 35d09ad1c61512e0da21d29f17cd9c696801398a17956b5de309369190f381e8: Status 404 returned error can't find the container with id 35d09ad1c61512e0da21d29f17cd9c696801398a17956b5de309369190f381e8
Oct 30 00:49:34.620775 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.620736    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:49:34.620920 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.620857    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:49:34.620994 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.620930    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:49:34.766433 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.766334    6099 generic.go:356] "Generic (PLEG): container finished" podID="46d85827-e8ab-4ca2-9e90-25dd27204143" containerID="2d9b4e149b15791982eff5a6e596f37fa0363eecd1d7b01172100e24473dc046" exitCode=0
Oct 30 00:49:34.766433 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.766425    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw" event={"ID":"46d85827-e8ab-4ca2-9e90-25dd27204143","Type":"ContainerDied","Data":"2d9b4e149b15791982eff5a6e596f37fa0363eecd1d7b01172100e24473dc046"}
Oct 30 00:49:34.768315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.768287    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl" event={"ID":"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999","Type":"ContainerStarted","Data":"00485a7d725182467e342da4f7072ea674d22ace36ac7e5c20acfe157bc6b74f"}
Oct 30 00:49:34.768467 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.768336    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl" event={"ID":"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999","Type":"ContainerStarted","Data":"35d09ad1c61512e0da21d29f17cd9c696801398a17956b5de309369190f381e8"}
Oct 30 00:49:34.845789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:34.845730    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl" podStartSLOduration=1.845697379 podStartE2EDuration="1.845697379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:34.8412907 +0000 UTC m=+868.698198457" watchObservedRunningTime="2026-10-30 00:49:34.845697379 +0000 UTC m=+868.702605137"
Oct 30 00:49:35.304213 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:35.304178    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:35.305744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:35.305716    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:49:35.305882 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:35.305765    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl" podUID="90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 00:49:35.734058 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:35.733980    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj"]
Oct 30 00:49:35.738615 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:35.738592    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj"
Oct 30 00:49:35.757124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:35.757098    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj"]
Oct 30 00:49:35.914550 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:35.914512    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ece9619-1e8f-479c-8202-787f87779fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13c4ad51-ctdqj\" (UID: \"9ece9619-1e8f-479c-8202-787f87779fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj"
Oct 30 00:49:35.915061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:35.914771    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z6p8\" (UniqueName: \"kubernetes.io/projected/9ece9619-1e8f-479c-8202-787f87779fdb-kube-api-access-9z6p8\") pod \"maas-api-revoke-sub-13c4ad51-ctdqj\" (UID: \"9ece9619-1e8f-479c-8202-787f87779fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj"
Oct 30 00:49:35.915061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:35.914807    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ece9619-1e8f-479c-8202-787f87779fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13c4ad51-ctdqj\" (UID: \"9ece9619-1e8f-479c-8202-787f87779fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj"
Oct 30 00:49:36.008211 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.008186    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw"
Oct 30 00:49:36.016528 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.016503    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z6p8\" (UniqueName: \"kubernetes.io/projected/9ece9619-1e8f-479c-8202-787f87779fdb-kube-api-access-9z6p8\") pod \"maas-api-revoke-sub-13c4ad51-ctdqj\" (UID: \"9ece9619-1e8f-479c-8202-787f87779fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj"
Oct 30 00:49:36.016651 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.016540    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ece9619-1e8f-479c-8202-787f87779fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13c4ad51-ctdqj\" (UID: \"9ece9619-1e8f-479c-8202-787f87779fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj"
Oct 30 00:49:36.016651 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.016604    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ece9619-1e8f-479c-8202-787f87779fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13c4ad51-ctdqj\" (UID: \"9ece9619-1e8f-479c-8202-787f87779fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj"
Oct 30 00:49:36.017320 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.017300    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ece9619-1e8f-479c-8202-787f87779fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13c4ad51-ctdqj\" (UID: \"9ece9619-1e8f-479c-8202-787f87779fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj"
Oct 30 00:49:36.018986 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.018965    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ece9619-1e8f-479c-8202-787f87779fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13c4ad51-ctdqj\" (UID: \"9ece9619-1e8f-479c-8202-787f87779fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj"
Oct 30 00:49:36.037795 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.037769    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z6p8\" (UniqueName: \"kubernetes.io/projected/9ece9619-1e8f-479c-8202-787f87779fdb-kube-api-access-9z6p8\") pod \"maas-api-revoke-sub-13c4ad51-ctdqj\" (UID: \"9ece9619-1e8f-479c-8202-787f87779fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj"
Oct 30 00:49:36.053299 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.052928    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj"
Oct 30 00:49:36.117341 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.117296    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d85827-e8ab-4ca2-9e90-25dd27204143-kube-api-access-6lxmm\" (UniqueName: \"kubernetes.io/projected/46d85827-e8ab-4ca2-9e90-25dd27204143-kube-api-access-6lxmm\") pod \"46d85827-e8ab-4ca2-9e90-25dd27204143\" (UID: \"46d85827-e8ab-4ca2-9e90-25dd27204143\") "
Oct 30 00:49:36.117503 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.117484    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46d85827-e8ab-4ca2-9e90-25dd27204143-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d85827-e8ab-4ca2-9e90-25dd27204143-maas-api-xks-ca\") pod \"46d85827-e8ab-4ca2-9e90-25dd27204143\" (UID: \"46d85827-e8ab-4ca2-9e90-25dd27204143\") "
Oct 30 00:49:36.117584 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.117566    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d85827-e8ab-4ca2-9e90-25dd27204143-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d85827-e8ab-4ca2-9e90-25dd27204143-maas-api-openshift-ca\") pod \"46d85827-e8ab-4ca2-9e90-25dd27204143\" (UID: \"46d85827-e8ab-4ca2-9e90-25dd27204143\") "
Oct 30 00:49:36.118215 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.118183    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d85827-e8ab-4ca2-9e90-25dd27204143-maas-api-openshift-ca" pod "46d85827-e8ab-4ca2-9e90-25dd27204143" (UID: "46d85827-e8ab-4ca2-9e90-25dd27204143"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:49:36.119642 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.119616    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d85827-e8ab-4ca2-9e90-25dd27204143-maas-api-xks-ca" pod "46d85827-e8ab-4ca2-9e90-25dd27204143" (UID: "46d85827-e8ab-4ca2-9e90-25dd27204143"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:49:36.125749 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.125726    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d85827-e8ab-4ca2-9e90-25dd27204143-kube-api-access-6lxmm" pod "46d85827-e8ab-4ca2-9e90-25dd27204143" (UID: "46d85827-e8ab-4ca2-9e90-25dd27204143"). InnerVolumeSpecName "kube-api-access-6lxmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:49:36.220010 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.219976    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lxmm\" (UniqueName: \"kubernetes.io/projected/46d85827-e8ab-4ca2-9e90-25dd27204143-kube-api-access-6lxmm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:36.220010 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.220003    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d85827-e8ab-4ca2-9e90-25dd27204143-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:36.220010 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.220013    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d85827-e8ab-4ca2-9e90-25dd27204143-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:36.303949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.303922    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:49:36.304075 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.303969    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl" podUID="90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 00:49:36.393959 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.393922    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj"]
Oct 30 00:49:36.394255 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:49:36.394230    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ece9619_1e8f_479c_8202_787f87779fdb.slice/crio-782c9527a1a08337f8493c258c6cebb1c25ef736e973322fc561ec9f7b8dde42 WatchSource:0}: Error finding container 782c9527a1a08337f8493c258c6cebb1c25ef736e973322fc561ec9f7b8dde42: Status 404 returned error can't find the container with id 782c9527a1a08337f8493c258c6cebb1c25ef736e973322fc561ec9f7b8dde42
Oct 30 00:49:36.818090 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.818059    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj" event={"ID":"9ece9619-1e8f-479c-8202-787f87779fdb","Type":"ContainerStarted","Data":"2044a3dd2f5b8444d561b25d943ac0f8c251e4d67ca7358d858ba30b299994cc"}
Oct 30 00:49:36.818269 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.818117    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj" event={"ID":"9ece9619-1e8f-479c-8202-787f87779fdb","Type":"ContainerStarted","Data":"782c9527a1a08337f8493c258c6cebb1c25ef736e973322fc561ec9f7b8dde42"}
Oct 30 00:49:36.819441 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.819420    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw"
Oct 30 00:49:36.819519 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.819463    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw" event={"ID":"46d85827-e8ab-4ca2-9e90-25dd27204143","Type":"ContainerDied","Data":"3500a1171f213c332a9ff266d69f325bead2078aa07ca71f69e273592b785590"}
Oct 30 00:49:36.819519 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.819508    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3500a1171f213c332a9ff266d69f325bead2078aa07ca71f69e273592b785590"
Oct 30 00:49:36.856904 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:36.856846    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj" podStartSLOduration=1.8568335679999999 podStartE2EDuration="1.856833568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:36.854695442 +0000 UTC m=+870.711603197" watchObservedRunningTime="2026-10-30 00:49:36.856833568 +0000 UTC m=+870.713741322"
Oct 30 00:49:37.303855 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:37.303825    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:49:37.304235 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:37.303879    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl" podUID="90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 00:49:37.827757 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:37.827720    6099 generic.go:356] "Generic (PLEG): container finished" podID="9ece9619-1e8f-479c-8202-787f87779fdb" containerID="2044a3dd2f5b8444d561b25d943ac0f8c251e4d67ca7358d858ba30b299994cc" exitCode=0
Oct 30 00:49:37.827947 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:37.827809    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj" event={"ID":"9ece9619-1e8f-479c-8202-787f87779fdb","Type":"ContainerDied","Data":"2044a3dd2f5b8444d561b25d943ac0f8c251e4d67ca7358d858ba30b299994cc"}
Oct 30 00:49:38.303923 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:38.303892    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:49:38.304306 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:38.303940    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl" podUID="90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 00:49:39.022827 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:39.022800    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj"
Oct 30 00:49:39.064367 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:39.064335    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ece9619-1e8f-479c-8202-787f87779fdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ece9619-1e8f-479c-8202-787f87779fdb-maas-api-xks-ca\") pod \"9ece9619-1e8f-479c-8202-787f87779fdb\" (UID: \"9ece9619-1e8f-479c-8202-787f87779fdb\") "
Oct 30 00:49:39.064531 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:39.064424    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ece9619-1e8f-479c-8202-787f87779fdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ece9619-1e8f-479c-8202-787f87779fdb-maas-api-openshift-ca\") pod \"9ece9619-1e8f-479c-8202-787f87779fdb\" (UID: \"9ece9619-1e8f-479c-8202-787f87779fdb\") "
Oct 30 00:49:39.075725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:39.064591    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ece9619-1e8f-479c-8202-787f87779fdb-kube-api-access-9z6p8\" (UniqueName: \"kubernetes.io/projected/9ece9619-1e8f-479c-8202-787f87779fdb-kube-api-access-9z6p8\") pod \"9ece9619-1e8f-479c-8202-787f87779fdb\" (UID: \"9ece9619-1e8f-479c-8202-787f87779fdb\") "
Oct 30 00:49:39.097773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:39.094265    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ece9619-1e8f-479c-8202-787f87779fdb-maas-api-xks-ca" pod "9ece9619-1e8f-479c-8202-787f87779fdb" (UID: "9ece9619-1e8f-479c-8202-787f87779fdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:49:39.097773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:39.094842    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ece9619-1e8f-479c-8202-787f87779fdb-maas-api-openshift-ca" pod "9ece9619-1e8f-479c-8202-787f87779fdb" (UID: "9ece9619-1e8f-479c-8202-787f87779fdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:49:39.097773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:39.095754    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ece9619-1e8f-479c-8202-787f87779fdb-kube-api-access-9z6p8" pod "9ece9619-1e8f-479c-8202-787f87779fdb" (UID: "9ece9619-1e8f-479c-8202-787f87779fdb"). InnerVolumeSpecName "kube-api-access-9z6p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:49:39.190446 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:39.190077    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ece9619-1e8f-479c-8202-787f87779fdb-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:39.190446 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:39.190115    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z6p8\" (UniqueName: \"kubernetes.io/projected/9ece9619-1e8f-479c-8202-787f87779fdb-kube-api-access-9z6p8\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:39.190446 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:39.190133    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ece9619-1e8f-479c-8202-787f87779fdb-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:39.304045 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:39.304011    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:49:39.304456 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:39.304067    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl" podUID="90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 00:49:39.842922 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:39.842882    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj" event={"ID":"9ece9619-1e8f-479c-8202-787f87779fdb","Type":"ContainerDied","Data":"782c9527a1a08337f8493c258c6cebb1c25ef736e973322fc561ec9f7b8dde42"}
Oct 30 00:49:39.843088 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:39.842920    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj"
Oct 30 00:49:39.843088 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:39.842935    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="782c9527a1a08337f8493c258c6cebb1c25ef736e973322fc561ec9f7b8dde42"
Oct 30 00:49:40.337334 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:40.337140    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:49:40.338084 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:40.338027    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl" podUID="90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 00:49:41.305120 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:41.305077    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 00:49:41.305551 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:41.305516    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl" podUID="90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 00:49:41.982693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:41.982657    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:42.003306 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:42.003269    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:42.925403 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:42.925368    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:49:43.344806 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:43.344772    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:49:43.345242 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:43.344826    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl" podUID="90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:43.721838 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:43.721755    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:43.722239 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:43.722209    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:43.723115 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:43.723097    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 00:49:46.036244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.036214    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn"]
Oct 30 00:49:46.036836 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.036820    6099 cpu_manager.go:414] "containerMap: removing container" podUID="9ece9619-1e8f-479c-8202-787f87779fdb" containerName="revoke-keys"
Oct 30 00:49:46.036896 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.036837    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ece9619-1e8f-479c-8202-787f87779fdb" containerName="revoke-keys"
Oct 30 00:49:46.036896 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.036876    6099 cpu_manager.go:414] "containerMap: removing container" podUID="46d85827-e8ab-4ca2-9e90-25dd27204143" containerName="revoke-keys"
Oct 30 00:49:46.036896 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.036881    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d85827-e8ab-4ca2-9e90-25dd27204143" containerName="revoke-keys"
Oct 30 00:49:46.037004 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.036975    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ece9619-1e8f-479c-8202-787f87779fdb" containerName="revoke-keys"
Oct 30 00:49:46.037004 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.036991    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d85827-e8ab-4ca2-9e90-25dd27204143" containerName="revoke-keys"
Oct 30 00:49:46.040844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.040829    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn"
Oct 30 00:49:46.063497 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.063474    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:49:46.063619 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.063496    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:49:46.063619 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.063497    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:49:46.068310 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.068286    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn"]
Oct 30 00:49:46.120635 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.120609    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e46913-e1f7-453a-a96a-ef57dee0a389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33fa5953-9r8hn\" (UID: \"d1e46913-e1f7-453a-a96a-ef57dee0a389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn"
Oct 30 00:49:46.120796 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.120767    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e46913-e1f7-453a-a96a-ef57dee0a389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33fa5953-9r8hn\" (UID: \"d1e46913-e1f7-453a-a96a-ef57dee0a389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn"
Oct 30 00:49:46.120888 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.120872    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d9fg\" (UniqueName: \"kubernetes.io/projected/d1e46913-e1f7-453a-a96a-ef57dee0a389-kube-api-access-2d9fg\") pod \"maas-api-revoke-sub-33fa5953-9r8hn\" (UID: \"d1e46913-e1f7-453a-a96a-ef57dee0a389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn"
Oct 30 00:49:46.153652 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.153627    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:49:46.221946 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.221918    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e46913-e1f7-453a-a96a-ef57dee0a389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33fa5953-9r8hn\" (UID: \"d1e46913-e1f7-453a-a96a-ef57dee0a389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn"
Oct 30 00:49:46.222071 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.222013    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d9fg\" (UniqueName: \"kubernetes.io/projected/d1e46913-e1f7-453a-a96a-ef57dee0a389-kube-api-access-2d9fg\") pod \"maas-api-revoke-sub-33fa5953-9r8hn\" (UID: \"d1e46913-e1f7-453a-a96a-ef57dee0a389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn"
Oct 30 00:49:46.222158 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.222142    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e46913-e1f7-453a-a96a-ef57dee0a389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33fa5953-9r8hn\" (UID: \"d1e46913-e1f7-453a-a96a-ef57dee0a389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn"
Oct 30 00:49:46.222803 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.222781    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e46913-e1f7-453a-a96a-ef57dee0a389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33fa5953-9r8hn\" (UID: \"d1e46913-e1f7-453a-a96a-ef57dee0a389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn"
Oct 30 00:49:46.224269 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.224245    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e46913-e1f7-453a-a96a-ef57dee0a389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33fa5953-9r8hn\" (UID: \"d1e46913-e1f7-453a-a96a-ef57dee0a389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn"
Oct 30 00:49:46.243138 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.243116    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d9fg\" (UniqueName: \"kubernetes.io/projected/d1e46913-e1f7-453a-a96a-ef57dee0a389-kube-api-access-2d9fg\") pod \"maas-api-revoke-sub-33fa5953-9r8hn\" (UID: \"d1e46913-e1f7-453a-a96a-ef57dee0a389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn"
Oct 30 00:49:46.354802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.354764    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn"
Oct 30 00:49:46.631449 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:49:46.631361    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:49:46.652246 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.652224    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn"]
Oct 30 00:49:46.653744 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:49:46.653692    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e46913_e1f7_453a_a96a_ef57dee0a389.slice/crio-d17fc6be65e1e8a7a5eb68c90dc56dcc6eb8a32d3f6ec9ac8267c27d26099bee WatchSource:0}: Error finding container d17fc6be65e1e8a7a5eb68c90dc56dcc6eb8a32d3f6ec9ac8267c27d26099bee: Status 404 returned error can't find the container with id d17fc6be65e1e8a7a5eb68c90dc56dcc6eb8a32d3f6ec9ac8267c27d26099bee
Oct 30 00:49:46.945615 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:46.945533    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn" event={"ID":"d1e46913-e1f7-453a-a96a-ef57dee0a389","Type":"ContainerStarted","Data":"d17fc6be65e1e8a7a5eb68c90dc56dcc6eb8a32d3f6ec9ac8267c27d26099bee"}
Oct 30 00:49:47.957758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:47.957363    6099 generic.go:356] "Generic (PLEG): container finished" podID="d1e46913-e1f7-453a-a96a-ef57dee0a389" containerID="c9ed2c543ba8c76871f1695fd8162454bb23f95dfc638962f2d3d203bd11726d" exitCode=0
Oct 30 00:49:47.957758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:47.957415    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn" event={"ID":"d1e46913-e1f7-453a-a96a-ef57dee0a389","Type":"ContainerDied","Data":"c9ed2c543ba8c76871f1695fd8162454bb23f95dfc638962f2d3d203bd11726d"}
Oct 30 00:49:49.142125 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:49.142089    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn"
Oct 30 00:49:49.267248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:49.267158    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1e46913-e1f7-453a-a96a-ef57dee0a389-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e46913-e1f7-453a-a96a-ef57dee0a389-maas-api-xks-ca\") pod \"d1e46913-e1f7-453a-a96a-ef57dee0a389\" (UID: \"d1e46913-e1f7-453a-a96a-ef57dee0a389\") "
Oct 30 00:49:49.267248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:49.267207    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e46913-e1f7-453a-a96a-ef57dee0a389-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e46913-e1f7-453a-a96a-ef57dee0a389-maas-api-openshift-ca\") pod \"d1e46913-e1f7-453a-a96a-ef57dee0a389\" (UID: \"d1e46913-e1f7-453a-a96a-ef57dee0a389\") "
Oct 30 00:49:49.267485 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:49.267255    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e46913-e1f7-453a-a96a-ef57dee0a389-kube-api-access-2d9fg\" (UniqueName: \"kubernetes.io/projected/d1e46913-e1f7-453a-a96a-ef57dee0a389-kube-api-access-2d9fg\") pod \"d1e46913-e1f7-453a-a96a-ef57dee0a389\" (UID: \"d1e46913-e1f7-453a-a96a-ef57dee0a389\") "
Oct 30 00:49:49.267674 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:49.267638    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e46913-e1f7-453a-a96a-ef57dee0a389-maas-api-openshift-ca" pod "d1e46913-e1f7-453a-a96a-ef57dee0a389" (UID: "d1e46913-e1f7-453a-a96a-ef57dee0a389"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:49:49.267905 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:49.267885    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e46913-e1f7-453a-a96a-ef57dee0a389-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:49.269306 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:49.269284    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e46913-e1f7-453a-a96a-ef57dee0a389-maas-api-xks-ca" pod "d1e46913-e1f7-453a-a96a-ef57dee0a389" (UID: "d1e46913-e1f7-453a-a96a-ef57dee0a389"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:49:49.269767 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:49.269748    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e46913-e1f7-453a-a96a-ef57dee0a389-kube-api-access-2d9fg" pod "d1e46913-e1f7-453a-a96a-ef57dee0a389" (UID: "d1e46913-e1f7-453a-a96a-ef57dee0a389"). InnerVolumeSpecName "kube-api-access-2d9fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:49:49.369080 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:49.369039    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e46913-e1f7-453a-a96a-ef57dee0a389-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:49.369080 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:49.369069    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d9fg\" (UniqueName: \"kubernetes.io/projected/d1e46913-e1f7-453a-a96a-ef57dee0a389-kube-api-access-2d9fg\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:49.971946 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:49.971915    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn"
Oct 30 00:49:49.971946 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:49.971935    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn" event={"ID":"d1e46913-e1f7-453a-a96a-ef57dee0a389","Type":"ContainerDied","Data":"d17fc6be65e1e8a7a5eb68c90dc56dcc6eb8a32d3f6ec9ac8267c27d26099bee"}
Oct 30 00:49:49.972191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:49.971987    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d17fc6be65e1e8a7a5eb68c90dc56dcc6eb8a32d3f6ec9ac8267c27d26099bee"
Oct 30 00:49:50.131045 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.131015    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x"]
Oct 30 00:49:50.131616 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.131595    6099 cpu_manager.go:414] "containerMap: removing container" podUID="d1e46913-e1f7-453a-a96a-ef57dee0a389" containerName="revoke-keys"
Oct 30 00:49:50.131616 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.131610    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e46913-e1f7-453a-a96a-ef57dee0a389" containerName="revoke-keys"
Oct 30 00:49:50.131823 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.131692    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e46913-e1f7-453a-a96a-ef57dee0a389" containerName="revoke-keys"
Oct 30 00:49:50.143320 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.143295    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4"]
Oct 30 00:49:50.144946 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.143854    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x"
Oct 30 00:49:50.162014 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.161990    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4"
Oct 30 00:49:50.162992 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.162725    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:49:50.163172 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.162833    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:49:50.164561 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.164536    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x"]
Oct 30 00:49:50.167664 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.167644    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4"]
Oct 30 00:49:50.168745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.168724    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:49:50.168837 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.168797    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:49:50.282827 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.282748    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d75d\" (UniqueName: \"kubernetes.io/projected/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-kube-api-access-9d75d\") pod \"maas-api-revoke-sub-f658383d-866h4\" (UID: \"6a5f5dc5-faba-4852-8639-5317f5cd1b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4"
Oct 30 00:49:50.282966 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.282868    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f658383d-866h4\" (UID: \"6a5f5dc5-faba-4852-8639-5317f5cd1b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4"
Oct 30 00:49:50.282966 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.282904    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09aabfe1-992e-40b3-b673-8f3d3af177a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b11644b3-qlb6x\" (UID: \"09aabfe1-992e-40b3-b673-8f3d3af177a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x"
Oct 30 00:49:50.282966 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.282959    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09aabfe1-992e-40b3-b673-8f3d3af177a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b11644b3-qlb6x\" (UID: \"09aabfe1-992e-40b3-b673-8f3d3af177a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x"
Oct 30 00:49:50.283086 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.283073    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r8wx\" (UniqueName: \"kubernetes.io/projected/09aabfe1-992e-40b3-b673-8f3d3af177a7-kube-api-access-2r8wx\") pod \"maas-api-revoke-sub-b11644b3-qlb6x\" (UID: \"09aabfe1-992e-40b3-b673-8f3d3af177a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x"
Oct 30 00:49:50.283119 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.283095    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f658383d-866h4\" (UID: \"6a5f5dc5-faba-4852-8639-5317f5cd1b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4"
Oct 30 00:49:50.384876 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.384844    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r8wx\" (UniqueName: \"kubernetes.io/projected/09aabfe1-992e-40b3-b673-8f3d3af177a7-kube-api-access-2r8wx\") pod \"maas-api-revoke-sub-b11644b3-qlb6x\" (UID: \"09aabfe1-992e-40b3-b673-8f3d3af177a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x"
Oct 30 00:49:50.384876 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.384879    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f658383d-866h4\" (UID: \"6a5f5dc5-faba-4852-8639-5317f5cd1b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4"
Oct 30 00:49:50.385097 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.384945    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d75d\" (UniqueName: \"kubernetes.io/projected/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-kube-api-access-9d75d\") pod \"maas-api-revoke-sub-f658383d-866h4\" (UID: \"6a5f5dc5-faba-4852-8639-5317f5cd1b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4"
Oct 30 00:49:50.385097 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.385011    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f658383d-866h4\" (UID: \"6a5f5dc5-faba-4852-8639-5317f5cd1b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4"
Oct 30 00:49:50.385097 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.385038    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09aabfe1-992e-40b3-b673-8f3d3af177a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b11644b3-qlb6x\" (UID: \"09aabfe1-992e-40b3-b673-8f3d3af177a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x"
Oct 30 00:49:50.385097 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.385067    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09aabfe1-992e-40b3-b673-8f3d3af177a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b11644b3-qlb6x\" (UID: \"09aabfe1-992e-40b3-b673-8f3d3af177a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x"
Oct 30 00:49:50.385661 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.385634    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f658383d-866h4\" (UID: \"6a5f5dc5-faba-4852-8639-5317f5cd1b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4"
Oct 30 00:49:50.385805 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.385678    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09aabfe1-992e-40b3-b673-8f3d3af177a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b11644b3-qlb6x\" (UID: \"09aabfe1-992e-40b3-b673-8f3d3af177a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x"
Oct 30 00:49:50.387332 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.387302    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09aabfe1-992e-40b3-b673-8f3d3af177a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b11644b3-qlb6x\" (UID: \"09aabfe1-992e-40b3-b673-8f3d3af177a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x"
Oct 30 00:49:50.387436 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.387411    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f658383d-866h4\" (UID: \"6a5f5dc5-faba-4852-8639-5317f5cd1b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4"
Oct 30 00:49:50.406627 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.406600    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d75d\" (UniqueName: \"kubernetes.io/projected/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-kube-api-access-9d75d\") pod \"maas-api-revoke-sub-f658383d-866h4\" (UID: \"6a5f5dc5-faba-4852-8639-5317f5cd1b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4"
Oct 30 00:49:50.406771 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.406754    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r8wx\" (UniqueName: \"kubernetes.io/projected/09aabfe1-992e-40b3-b673-8f3d3af177a7-kube-api-access-2r8wx\") pod \"maas-api-revoke-sub-b11644b3-qlb6x\" (UID: \"09aabfe1-992e-40b3-b673-8f3d3af177a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x"
Oct 30 00:49:50.467977 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.467952    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x"
Oct 30 00:49:50.480342 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.480320    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4"
Oct 30 00:49:50.778659 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.778634    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x"]
Oct 30 00:49:50.779662 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:49:50.779626    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09aabfe1_992e_40b3_b673_8f3d3af177a7.slice/crio-18c3dba1a331e3b567da605eaa40f2c25e054101dc44194664ab46ac2e321db7 WatchSource:0}: Error finding container 18c3dba1a331e3b567da605eaa40f2c25e054101dc44194664ab46ac2e321db7: Status 404 returned error can't find the container with id 18c3dba1a331e3b567da605eaa40f2c25e054101dc44194664ab46ac2e321db7
Oct 30 00:49:50.789547 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.789523    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4"]
Oct 30 00:49:50.791812 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:49:50.791790    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5f5dc5_faba_4852_8639_5317f5cd1b7d.slice/crio-bcccbab3f652e12f27e19be1b2d716ceeda7062ccc2b4382491256d4ce41877f WatchSource:0}: Error finding container bcccbab3f652e12f27e19be1b2d716ceeda7062ccc2b4382491256d4ce41877f: Status 404 returned error can't find the container with id bcccbab3f652e12f27e19be1b2d716ceeda7062ccc2b4382491256d4ce41877f
Oct 30 00:49:50.981249 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.981217    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x" event={"ID":"09aabfe1-992e-40b3-b673-8f3d3af177a7","Type":"ContainerStarted","Data":"18c3dba1a331e3b567da605eaa40f2c25e054101dc44194664ab46ac2e321db7"}
Oct 30 00:49:50.982432 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:50.982408    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4" event={"ID":"6a5f5dc5-faba-4852-8639-5317f5cd1b7d","Type":"ContainerStarted","Data":"bcccbab3f652e12f27e19be1b2d716ceeda7062ccc2b4382491256d4ce41877f"}
Oct 30 00:49:51.990744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:51.990649    6099 generic.go:356] "Generic (PLEG): container finished" podID="6a5f5dc5-faba-4852-8639-5317f5cd1b7d" containerID="d7826c4b4e8a2fdcc4d4eb84720b5b9e81d4a694f0df7ce905630deb756cb2bb" exitCode=0
Oct 30 00:49:51.990744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:51.990733    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4" event={"ID":"6a5f5dc5-faba-4852-8639-5317f5cd1b7d","Type":"ContainerDied","Data":"d7826c4b4e8a2fdcc4d4eb84720b5b9e81d4a694f0df7ce905630deb756cb2bb"}
Oct 30 00:49:51.992493 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:51.992468    6099 generic.go:356] "Generic (PLEG): container finished" podID="09aabfe1-992e-40b3-b673-8f3d3af177a7" containerID="146bad53f9092afa1d86922a2e10956eed4e3ec2923c837b1f39c39a396f40d8" exitCode=0
Oct 30 00:49:51.992599 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:51.992500    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x" event={"ID":"09aabfe1-992e-40b3-b673-8f3d3af177a7","Type":"ContainerDied","Data":"146bad53f9092afa1d86922a2e10956eed4e3ec2923c837b1f39c39a396f40d8"}
Oct 30 00:49:53.171846 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.171814    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4"
Oct 30 00:49:53.211981 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.211953    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x"
Oct 30 00:49:53.222989 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.222961    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-maas-api-openshift-ca\") pod \"6a5f5dc5-faba-4852-8639-5317f5cd1b7d\" (UID: \"6a5f5dc5-faba-4852-8639-5317f5cd1b7d\") "
Oct 30 00:49:53.223157 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.222997    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-kube-api-access-9d75d\" (UniqueName: \"kubernetes.io/projected/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-kube-api-access-9d75d\") pod \"6a5f5dc5-faba-4852-8639-5317f5cd1b7d\" (UID: \"6a5f5dc5-faba-4852-8639-5317f5cd1b7d\") "
Oct 30 00:49:53.223157 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.223014    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09aabfe1-992e-40b3-b673-8f3d3af177a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09aabfe1-992e-40b3-b673-8f3d3af177a7-maas-api-openshift-ca\") pod \"09aabfe1-992e-40b3-b673-8f3d3af177a7\" (UID: \"09aabfe1-992e-40b3-b673-8f3d3af177a7\") "
Oct 30 00:49:53.223157 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.223144    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09aabfe1-992e-40b3-b673-8f3d3af177a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09aabfe1-992e-40b3-b673-8f3d3af177a7-maas-api-xks-ca\") pod \"09aabfe1-992e-40b3-b673-8f3d3af177a7\" (UID: \"09aabfe1-992e-40b3-b673-8f3d3af177a7\") "
Oct 30 00:49:53.223431 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.223390    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09aabfe1-992e-40b3-b673-8f3d3af177a7-maas-api-openshift-ca" pod "09aabfe1-992e-40b3-b673-8f3d3af177a7" (UID: "09aabfe1-992e-40b3-b673-8f3d3af177a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:49:53.223431 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.223401    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-maas-api-openshift-ca" pod "6a5f5dc5-faba-4852-8639-5317f5cd1b7d" (UID: "6a5f5dc5-faba-4852-8639-5317f5cd1b7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:49:53.223915 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.223890    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:53.224012 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.223919    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09aabfe1-992e-40b3-b673-8f3d3af177a7-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:53.225741 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.225686    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09aabfe1-992e-40b3-b673-8f3d3af177a7-maas-api-xks-ca" pod "09aabfe1-992e-40b3-b673-8f3d3af177a7" (UID: "09aabfe1-992e-40b3-b673-8f3d3af177a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:49:53.226848 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.226828    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-kube-api-access-9d75d" pod "6a5f5dc5-faba-4852-8639-5317f5cd1b7d" (UID: "6a5f5dc5-faba-4852-8639-5317f5cd1b7d"). InnerVolumeSpecName "kube-api-access-9d75d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:49:53.325111 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.325080    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09aabfe1-992e-40b3-b673-8f3d3af177a7-kube-api-access-2r8wx\" (UniqueName: \"kubernetes.io/projected/09aabfe1-992e-40b3-b673-8f3d3af177a7-kube-api-access-2r8wx\") pod \"09aabfe1-992e-40b3-b673-8f3d3af177a7\" (UID: \"09aabfe1-992e-40b3-b673-8f3d3af177a7\") "
Oct 30 00:49:53.325277 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.325148    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-maas-api-xks-ca\") pod \"6a5f5dc5-faba-4852-8639-5317f5cd1b7d\" (UID: \"6a5f5dc5-faba-4852-8639-5317f5cd1b7d\") "
Oct 30 00:49:53.325645 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.325623    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d75d\" (UniqueName: \"kubernetes.io/projected/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-kube-api-access-9d75d\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:53.325645 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.325644    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09aabfe1-992e-40b3-b673-8f3d3af177a7-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:53.327165 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.327139    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-maas-api-xks-ca" pod "6a5f5dc5-faba-4852-8639-5317f5cd1b7d" (UID: "6a5f5dc5-faba-4852-8639-5317f5cd1b7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:49:53.327248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.327189    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09aabfe1-992e-40b3-b673-8f3d3af177a7-kube-api-access-2r8wx" pod "09aabfe1-992e-40b3-b673-8f3d3af177a7" (UID: "09aabfe1-992e-40b3-b673-8f3d3af177a7"). InnerVolumeSpecName "kube-api-access-2r8wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:49:53.426525 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.426498    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5f5dc5-faba-4852-8639-5317f5cd1b7d-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:53.426525 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:53.426521    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r8wx\" (UniqueName: \"kubernetes.io/projected/09aabfe1-992e-40b3-b673-8f3d3af177a7-kube-api-access-2r8wx\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:49:54.014776 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:54.014740    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4" event={"ID":"6a5f5dc5-faba-4852-8639-5317f5cd1b7d","Type":"ContainerDied","Data":"bcccbab3f652e12f27e19be1b2d716ceeda7062ccc2b4382491256d4ce41877f"}
Oct 30 00:49:54.014966 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:54.014796    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcccbab3f652e12f27e19be1b2d716ceeda7062ccc2b4382491256d4ce41877f"
Oct 30 00:49:54.014966 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:54.014749    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4"
Oct 30 00:49:54.016256 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:54.016235    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x"
Oct 30 00:49:54.016378 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:54.016268    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x" event={"ID":"09aabfe1-992e-40b3-b673-8f3d3af177a7","Type":"ContainerDied","Data":"18c3dba1a331e3b567da605eaa40f2c25e054101dc44194664ab46ac2e321db7"}
Oct 30 00:49:54.016378 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:54.016301    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18c3dba1a331e3b567da605eaa40f2c25e054101dc44194664ab46ac2e321db7"
Oct 30 00:49:56.422373 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.422339    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94"]
Oct 30 00:49:56.422923 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.422909    6099 cpu_manager.go:414] "containerMap: removing container" podUID="09aabfe1-992e-40b3-b673-8f3d3af177a7" containerName="revoke-keys"
Oct 30 00:49:56.422964 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.422923    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09aabfe1-992e-40b3-b673-8f3d3af177a7" containerName="revoke-keys"
Oct 30 00:49:56.422964 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.422950    6099 cpu_manager.go:414] "containerMap: removing container" podUID="6a5f5dc5-faba-4852-8639-5317f5cd1b7d" containerName="revoke-keys"
Oct 30 00:49:56.422964 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.422955    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5f5dc5-faba-4852-8639-5317f5cd1b7d" containerName="revoke-keys"
Oct 30 00:49:56.423071 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.423060    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="09aabfe1-992e-40b3-b673-8f3d3af177a7" containerName="revoke-keys"
Oct 30 00:49:56.423105 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.423073    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5f5dc5-faba-4852-8639-5317f5cd1b7d" containerName="revoke-keys"
Oct 30 00:49:56.426949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.426931    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94"
Oct 30 00:49:56.456752 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.452802    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:49:56.456752 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.452903    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:49:56.456752 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.453501    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:49:56.458394 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.458363    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94"]
Oct 30 00:49:56.458594 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.458575    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:49:56.466984 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.466955    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed772a4d-cnt94\" (UID: \"6de33c43-4c4f-4293-9ab8-afe9ccd961d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94"
Oct 30 00:49:56.467088 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.467029    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k8rq\" (UniqueName: \"kubernetes.io/projected/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-kube-api-access-6k8rq\") pod \"maas-api-revoke-sub-ed772a4d-cnt94\" (UID: \"6de33c43-4c4f-4293-9ab8-afe9ccd961d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94"
Oct 30 00:49:56.467193 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.467174    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed772a4d-cnt94\" (UID: \"6de33c43-4c4f-4293-9ab8-afe9ccd961d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94"
Oct 30 00:49:56.568406 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.568374    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed772a4d-cnt94\" (UID: \"6de33c43-4c4f-4293-9ab8-afe9ccd961d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94"
Oct 30 00:49:56.568593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.568441    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k8rq\" (UniqueName: \"kubernetes.io/projected/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-kube-api-access-6k8rq\") pod \"maas-api-revoke-sub-ed772a4d-cnt94\" (UID: \"6de33c43-4c4f-4293-9ab8-afe9ccd961d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94"
Oct 30 00:49:56.568593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.568525    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed772a4d-cnt94\" (UID: \"6de33c43-4c4f-4293-9ab8-afe9ccd961d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94"
Oct 30 00:49:56.569107 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.569070    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed772a4d-cnt94\" (UID: \"6de33c43-4c4f-4293-9ab8-afe9ccd961d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94"
Oct 30 00:49:56.570780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.570759    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed772a4d-cnt94\" (UID: \"6de33c43-4c4f-4293-9ab8-afe9ccd961d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94"
Oct 30 00:49:56.589450 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.589418    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k8rq\" (UniqueName: \"kubernetes.io/projected/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-kube-api-access-6k8rq\") pod \"maas-api-revoke-sub-ed772a4d-cnt94\" (UID: \"6de33c43-4c4f-4293-9ab8-afe9ccd961d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94"
Oct 30 00:49:56.741163 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.741085    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94"
Oct 30 00:49:56.876675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:56.876645    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:49:57.072727 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:49:57.071444    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de33c43_4c4f_4293_9ab8_afe9ccd961d3.slice/crio-24b66c3f350916024b6363bd9b048c645511225b3b33d7bbe5ece1e30e88c3c6 WatchSource:0}: Error finding container 24b66c3f350916024b6363bd9b048c645511225b3b33d7bbe5ece1e30e88c3c6: Status 404 returned error can't find the container with id 24b66c3f350916024b6363bd9b048c645511225b3b33d7bbe5ece1e30e88c3c6
Oct 30 00:49:57.084734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:57.084141    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94"]
Oct 30 00:49:57.948491 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:57.944080    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks"]
Oct 30 00:49:57.977336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:57.949370    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks"]
Oct 30 00:49:57.977336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:57.949490    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks"
Oct 30 00:49:57.999845 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:57.997252    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlfrq\" (UniqueName: \"kubernetes.io/projected/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-kube-api-access-zlfrq\") pod \"maas-api-revoke-sub-8a89488a-mpjks\" (UID: \"67e6c73a-6b09-46fe-8e70-93bbf68c41bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks"
Oct 30 00:49:57.999845 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:57.997435    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a89488a-mpjks\" (UID: \"67e6c73a-6b09-46fe-8e70-93bbf68c41bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks"
Oct 30 00:49:57.999845 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:57.997538    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a89488a-mpjks\" (UID: \"67e6c73a-6b09-46fe-8e70-93bbf68c41bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks"
Oct 30 00:49:58.054896 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:58.054846    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94" event={"ID":"6de33c43-4c4f-4293-9ab8-afe9ccd961d3","Type":"ContainerStarted","Data":"9ce3278be7c6f8b318b0268176c7c5083ea609e6700d27826dc25277db9e5cec"}
Oct 30 00:49:58.055204 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:58.055147    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94" event={"ID":"6de33c43-4c4f-4293-9ab8-afe9ccd961d3","Type":"ContainerStarted","Data":"24b66c3f350916024b6363bd9b048c645511225b3b33d7bbe5ece1e30e88c3c6"}
Oct 30 00:49:58.108737 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:58.106365    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlfrq\" (UniqueName: \"kubernetes.io/projected/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-kube-api-access-zlfrq\") pod \"maas-api-revoke-sub-8a89488a-mpjks\" (UID: \"67e6c73a-6b09-46fe-8e70-93bbf68c41bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks"
Oct 30 00:49:58.108737 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:58.106587    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a89488a-mpjks\" (UID: \"67e6c73a-6b09-46fe-8e70-93bbf68c41bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks"
Oct 30 00:49:58.108737 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:58.106699    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a89488a-mpjks\" (UID: \"67e6c73a-6b09-46fe-8e70-93bbf68c41bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks"
Oct 30 00:49:58.108737 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:58.108661    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a89488a-mpjks\" (UID: \"67e6c73a-6b09-46fe-8e70-93bbf68c41bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks"
Oct 30 00:49:58.121728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:58.112097    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a89488a-mpjks\" (UID: \"67e6c73a-6b09-46fe-8e70-93bbf68c41bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks"
Oct 30 00:49:58.145938 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:58.143167    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlfrq\" (UniqueName: \"kubernetes.io/projected/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-kube-api-access-zlfrq\") pod \"maas-api-revoke-sub-8a89488a-mpjks\" (UID: \"67e6c73a-6b09-46fe-8e70-93bbf68c41bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks"
Oct 30 00:49:58.340731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:58.339693    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks"
Oct 30 00:49:59.319730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:59.312274    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:59.610853 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:59.579136    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 00:49:59.610853 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:59.603925    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 00:49:59.656863 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:59.656824    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:49:59.660947 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:59.660910    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:59.663373 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:59.663327    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94" podStartSLOduration=3.663314942 podStartE2EDuration="3.663314942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:58.10964953 +0000 UTC m=+891.966557286" watchObservedRunningTime="2026-10-30 00:49:59.663314942 +0000 UTC m=+893.520222697"
Oct 30 00:49:59.664634 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:59.664611    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks"]
Oct 30 00:49:59.715735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:59.689844    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:59.757789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:59.744452    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:49:59.858021 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:59.857396    6099 generic.go:356] "Generic (PLEG): container finished" podID="6de33c43-4c4f-4293-9ab8-afe9ccd961d3" containerID="9ce3278be7c6f8b318b0268176c7c5083ea609e6700d27826dc25277db9e5cec" exitCode=0
Oct 30 00:49:59.858021 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:49:59.857599    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94" event={"ID":"6de33c43-4c4f-4293-9ab8-afe9ccd961d3","Type":"ContainerDied","Data":"9ce3278be7c6f8b318b0268176c7c5083ea609e6700d27826dc25277db9e5cec"}
Oct 30 00:50:00.866575 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:00.866534    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks" event={"ID":"67e6c73a-6b09-46fe-8e70-93bbf68c41bf","Type":"ContainerStarted","Data":"24b83bf894ee166f3e654d5fc6ebb3ef66e43b163f7cd510fb940d4dbe073861"}
Oct 30 00:50:00.867062 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:00.866605    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks" event={"ID":"67e6c73a-6b09-46fe-8e70-93bbf68c41bf","Type":"ContainerStarted","Data":"905f5c071d05632dd4bb4d5a2df8abd3f797484347d0a6b555d242a52700e2a1"}
Oct 30 00:50:00.904941 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:00.904891    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks" podStartSLOduration=3.9048780880000002 podStartE2EDuration="3.904878088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:00.90373543 +0000 UTC m=+894.760643186" watchObservedRunningTime="2026-10-30 00:50:00.904878088 +0000 UTC m=+894.761785842"
Oct 30 00:50:01.025895 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:01.025865    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94"
Oct 30 00:50:01.223808 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:01.223785    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-kube-api-access-6k8rq\" (UniqueName: \"kubernetes.io/projected/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-kube-api-access-6k8rq\") pod \"6de33c43-4c4f-4293-9ab8-afe9ccd961d3\" (UID: \"6de33c43-4c4f-4293-9ab8-afe9ccd961d3\") "
Oct 30 00:50:01.223913 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:01.223844    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-maas-api-openshift-ca\") pod \"6de33c43-4c4f-4293-9ab8-afe9ccd961d3\" (UID: \"6de33c43-4c4f-4293-9ab8-afe9ccd961d3\") "
Oct 30 00:50:01.223913 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:01.223866    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-maas-api-xks-ca\") pod \"6de33c43-4c4f-4293-9ab8-afe9ccd961d3\" (UID: \"6de33c43-4c4f-4293-9ab8-afe9ccd961d3\") "
Oct 30 00:50:01.224231 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:01.224204    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-maas-api-openshift-ca" pod "6de33c43-4c4f-4293-9ab8-afe9ccd961d3" (UID: "6de33c43-4c4f-4293-9ab8-afe9ccd961d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:01.224540 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:01.224523    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:01.225831 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:01.225801    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-maas-api-xks-ca" pod "6de33c43-4c4f-4293-9ab8-afe9ccd961d3" (UID: "6de33c43-4c4f-4293-9ab8-afe9ccd961d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:50:01.225969 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:01.225952    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-kube-api-access-6k8rq" pod "6de33c43-4c4f-4293-9ab8-afe9ccd961d3" (UID: "6de33c43-4c4f-4293-9ab8-afe9ccd961d3"). InnerVolumeSpecName "kube-api-access-6k8rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:01.325842 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:01.325804    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:01.325842 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:01.325838    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k8rq\" (UniqueName: \"kubernetes.io/projected/6de33c43-4c4f-4293-9ab8-afe9ccd961d3-kube-api-access-6k8rq\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:01.876063 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:01.876030    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94"
Oct 30 00:50:01.876576 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:01.876117    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94" event={"ID":"6de33c43-4c4f-4293-9ab8-afe9ccd961d3","Type":"ContainerDied","Data":"24b66c3f350916024b6363bd9b048c645511225b3b33d7bbe5ece1e30e88c3c6"}
Oct 30 00:50:01.876576 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:01.876173    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24b66c3f350916024b6363bd9b048c645511225b3b33d7bbe5ece1e30e88c3c6"
Oct 30 00:50:01.882037 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:01.881995    6099 generic.go:356] "Generic (PLEG): container finished" podID="67e6c73a-6b09-46fe-8e70-93bbf68c41bf" containerID="24b83bf894ee166f3e654d5fc6ebb3ef66e43b163f7cd510fb940d4dbe073861" exitCode=0
Oct 30 00:50:01.882248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:01.882229    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks" event={"ID":"67e6c73a-6b09-46fe-8e70-93bbf68c41bf","Type":"ContainerDied","Data":"24b83bf894ee166f3e654d5fc6ebb3ef66e43b163f7cd510fb940d4dbe073861"}
Oct 30 00:50:03.046950 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.046922    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks"
Oct 30 00:50:03.147155 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.147087    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-kube-api-access-zlfrq\" (UniqueName: \"kubernetes.io/projected/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-kube-api-access-zlfrq\") pod \"67e6c73a-6b09-46fe-8e70-93bbf68c41bf\" (UID: \"67e6c73a-6b09-46fe-8e70-93bbf68c41bf\") "
Oct 30 00:50:03.147308 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.147287    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-maas-api-xks-ca\") pod \"67e6c73a-6b09-46fe-8e70-93bbf68c41bf\" (UID: \"67e6c73a-6b09-46fe-8e70-93bbf68c41bf\") "
Oct 30 00:50:03.147401 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.147384    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-maas-api-openshift-ca\") pod \"67e6c73a-6b09-46fe-8e70-93bbf68c41bf\" (UID: \"67e6c73a-6b09-46fe-8e70-93bbf68c41bf\") "
Oct 30 00:50:03.147788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.147756    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-maas-api-openshift-ca" pod "67e6c73a-6b09-46fe-8e70-93bbf68c41bf" (UID: "67e6c73a-6b09-46fe-8e70-93bbf68c41bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:03.147905 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.147887    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:03.149234 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.149210    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-kube-api-access-zlfrq" pod "67e6c73a-6b09-46fe-8e70-93bbf68c41bf" (UID: "67e6c73a-6b09-46fe-8e70-93bbf68c41bf"). InnerVolumeSpecName "kube-api-access-zlfrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:03.149324 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.149284    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-maas-api-xks-ca" pod "67e6c73a-6b09-46fe-8e70-93bbf68c41bf" (UID: "67e6c73a-6b09-46fe-8e70-93bbf68c41bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:50:03.249389 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.249357    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:03.249389 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.249386    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlfrq\" (UniqueName: \"kubernetes.io/projected/67e6c73a-6b09-46fe-8e70-93bbf68c41bf-kube-api-access-zlfrq\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:03.548199 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.548162    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"]
Oct 30 00:50:03.548832 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.548813    6099 cpu_manager.go:414] "containerMap: removing container" podUID="67e6c73a-6b09-46fe-8e70-93bbf68c41bf" containerName="revoke-keys"
Oct 30 00:50:03.548832 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.548832    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e6c73a-6b09-46fe-8e70-93bbf68c41bf" containerName="revoke-keys"
Oct 30 00:50:03.548958 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.548876    6099 cpu_manager.go:414] "containerMap: removing container" podUID="6de33c43-4c4f-4293-9ab8-afe9ccd961d3" containerName="revoke-keys"
Oct 30 00:50:03.548958 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.548881    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de33c43-4c4f-4293-9ab8-afe9ccd961d3" containerName="revoke-keys"
Oct 30 00:50:03.549023 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.548997    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de33c43-4c4f-4293-9ab8-afe9ccd961d3" containerName="revoke-keys"
Oct 30 00:50:03.549023 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.549012    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e6c73a-6b09-46fe-8e70-93bbf68c41bf" containerName="revoke-keys"
Oct 30 00:50:03.552800 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.552780    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.576854 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.576828    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"]
Oct 30 00:50:03.654310 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.654274    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.654310 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.654310    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.654502 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.654399    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.654543 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.654514    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.654584 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.654563    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.654626 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.654587    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.654737 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.654684    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.654846 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.654791    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxqp2\" (UniqueName: \"kubernetes.io/projected/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-kube-api-access-xxqp2\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.654846 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.654832    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.756320 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.756283    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.756512 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.756391    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.756512 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.756437    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.756512 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.756476    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.756685 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.756537    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.756685 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.756590    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxqp2\" (UniqueName: \"kubernetes.io/projected/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-kube-api-access-xxqp2\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.756685 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.756631    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.756860 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.756817    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.756860 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.756844    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.756967 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.756864    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.757155 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.757130    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.757250 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.757231    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.757427 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.757396    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.757537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.757447    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.759218 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.759190    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.759668 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.759645    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.778494 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.778445    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.778605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.778492    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxqp2\" (UniqueName: \"kubernetes.io/projected/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-kube-api-access-xxqp2\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.867600 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.867536    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:03.905215 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.905185    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks"
Oct 30 00:50:03.905356 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.905216    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks" event={"ID":"67e6c73a-6b09-46fe-8e70-93bbf68c41bf","Type":"ContainerDied","Data":"905f5c071d05632dd4bb4d5a2df8abd3f797484347d0a6b555d242a52700e2a1"}
Oct 30 00:50:03.905356 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:03.905266    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="905f5c071d05632dd4bb4d5a2df8abd3f797484347d0a6b555d242a52700e2a1"
Oct 30 00:50:04.182278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:04.182252    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"]
Oct 30 00:50:04.183782 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:50:04.183757    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbccdfc4_450a_4048_9f3b_f52be1b3bb22.slice/crio-dd6bc0b5aba47383ce0f97d405aefc3f9acb3d06ad27eb71f487d3f55ec289a6 WatchSource:0}: Error finding container dd6bc0b5aba47383ce0f97d405aefc3f9acb3d06ad27eb71f487d3f55ec289a6: Status 404 returned error can't find the container with id dd6bc0b5aba47383ce0f97d405aefc3f9acb3d06ad27eb71f487d3f55ec289a6
Oct 30 00:50:04.186273 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:04.186236    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:50:04.186378 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:04.186316    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:50:04.186378 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:04.186361    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:50:04.915397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:04.915358    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv" event={"ID":"cbccdfc4-450a-4048-9f3b-f52be1b3bb22","Type":"ContainerStarted","Data":"62aa0933c8a17805ab731007a95af3976b838e5a6623193ad67ba15c7049225b"}
Oct 30 00:50:04.915600 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:04.915423    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv" event={"ID":"cbccdfc4-450a-4048-9f3b-f52be1b3bb22","Type":"ContainerStarted","Data":"dd6bc0b5aba47383ce0f97d405aefc3f9acb3d06ad27eb71f487d3f55ec289a6"}
Oct 30 00:50:04.957638 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:04.957566    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv" podStartSLOduration=1.957551203 podStartE2EDuration="1.957551203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:04.955502405 +0000 UTC m=+898.812410160" watchObservedRunningTime="2026-10-30 00:50:04.957551203 +0000 UTC m=+898.814458958"
Oct 30 00:50:05.874934 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:05.874900    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:05.890876 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:05.890840    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:50:05.891126 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:05.891095    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv" podUID="cbccdfc4-450a-4048-9f3b-f52be1b3bb22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 00:50:06.400617 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:06.400583    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 00:50:06.405958 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:06.405936    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 00:50:06.889830 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:06.889791    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:50:06.890190 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:06.889853    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv" podUID="cbccdfc4-450a-4048-9f3b-f52be1b3bb22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 00:50:07.877843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:07.877804    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:50:07.878062 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:07.877868    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv" podUID="cbccdfc4-450a-4048-9f3b-f52be1b3bb22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 00:50:08.875735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:08.870017    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:50:08.875735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:08.870087    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv" podUID="cbccdfc4-450a-4048-9f3b-f52be1b3bb22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 00:50:09.876025 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:09.875988    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 00:50:09.876599 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:09.876567    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv" podUID="cbccdfc4-450a-4048-9f3b-f52be1b3bb22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 00:50:10.871660 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:10.871629    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:10.872159 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:10.872133    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:10.872779 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:10.872755    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:50:12.110876 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.110847    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s"]
Oct 30 00:50:12.116993 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.116972    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s"
Oct 30 00:50:12.129745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.129723    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:50:12.139825 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.139803    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:50:12.139915 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.139852    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:50:12.139915 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.139876    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:50:12.145604 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.145573    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s"]
Oct 30 00:50:12.189546 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.189518    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a997e50-2b16-4fab-8556-138de2d1cc11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-370faf19-g5t5s\" (UID: \"1a997e50-2b16-4fab-8556-138de2d1cc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s"
Oct 30 00:50:12.189774 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.189755    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6rvg\" (UniqueName: \"kubernetes.io/projected/1a997e50-2b16-4fab-8556-138de2d1cc11-kube-api-access-s6rvg\") pod \"maas-api-revoke-sub-370faf19-g5t5s\" (UID: \"1a997e50-2b16-4fab-8556-138de2d1cc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s"
Oct 30 00:50:12.189825 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.189807    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a997e50-2b16-4fab-8556-138de2d1cc11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-370faf19-g5t5s\" (UID: \"1a997e50-2b16-4fab-8556-138de2d1cc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s"
Oct 30 00:50:12.291022 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.290996    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6rvg\" (UniqueName: \"kubernetes.io/projected/1a997e50-2b16-4fab-8556-138de2d1cc11-kube-api-access-s6rvg\") pod \"maas-api-revoke-sub-370faf19-g5t5s\" (UID: \"1a997e50-2b16-4fab-8556-138de2d1cc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s"
Oct 30 00:50:12.291185 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.291044    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a997e50-2b16-4fab-8556-138de2d1cc11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-370faf19-g5t5s\" (UID: \"1a997e50-2b16-4fab-8556-138de2d1cc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s"
Oct 30 00:50:12.291185 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.291115    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a997e50-2b16-4fab-8556-138de2d1cc11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-370faf19-g5t5s\" (UID: \"1a997e50-2b16-4fab-8556-138de2d1cc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s"
Oct 30 00:50:12.291701 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.291666    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a997e50-2b16-4fab-8556-138de2d1cc11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-370faf19-g5t5s\" (UID: \"1a997e50-2b16-4fab-8556-138de2d1cc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s"
Oct 30 00:50:12.293416 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.293396    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a997e50-2b16-4fab-8556-138de2d1cc11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-370faf19-g5t5s\" (UID: \"1a997e50-2b16-4fab-8556-138de2d1cc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s"
Oct 30 00:50:12.312396 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.312376    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6rvg\" (UniqueName: \"kubernetes.io/projected/1a997e50-2b16-4fab-8556-138de2d1cc11-kube-api-access-s6rvg\") pod \"maas-api-revoke-sub-370faf19-g5t5s\" (UID: \"1a997e50-2b16-4fab-8556-138de2d1cc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s"
Oct 30 00:50:12.430740 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.430650    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s"
Oct 30 00:50:12.728648 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.728616    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s"]
Oct 30 00:50:12.729401 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:50:12.729364    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a997e50_2b16_4fab_8556_138de2d1cc11.slice/crio-f81868553fdae6c280809445282e5c38ad4e7dcbc3070a9219343edd029052a0 WatchSource:0}: Error finding container f81868553fdae6c280809445282e5c38ad4e7dcbc3070a9219343edd029052a0: Status 404 returned error can't find the container with id f81868553fdae6c280809445282e5c38ad4e7dcbc3070a9219343edd029052a0
Oct 30 00:50:12.979276 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:12.979249    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s" event={"ID":"1a997e50-2b16-4fab-8556-138de2d1cc11","Type":"ContainerStarted","Data":"f81868553fdae6c280809445282e5c38ad4e7dcbc3070a9219343edd029052a0"}
Oct 30 00:50:13.987464 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:13.987381    6099 generic.go:356] "Generic (PLEG): container finished" podID="1a997e50-2b16-4fab-8556-138de2d1cc11" containerID="5706c6a24e324265d6c19123e14716de9bf82d9af004ca6749a0adb4fad2b3e3" exitCode=0
Oct 30 00:50:13.987464 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:13.987423    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s" event={"ID":"1a997e50-2b16-4fab-8556-138de2d1cc11","Type":"ContainerDied","Data":"5706c6a24e324265d6c19123e14716de9bf82d9af004ca6749a0adb4fad2b3e3"}
Oct 30 00:50:15.141491 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:15.141464    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s"
Oct 30 00:50:15.232523 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:15.232486    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a997e50-2b16-4fab-8556-138de2d1cc11-kube-api-access-s6rvg\" (UniqueName: \"kubernetes.io/projected/1a997e50-2b16-4fab-8556-138de2d1cc11-kube-api-access-s6rvg\") pod \"1a997e50-2b16-4fab-8556-138de2d1cc11\" (UID: \"1a997e50-2b16-4fab-8556-138de2d1cc11\") "
Oct 30 00:50:15.232688 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:15.232674    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a997e50-2b16-4fab-8556-138de2d1cc11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a997e50-2b16-4fab-8556-138de2d1cc11-maas-api-xks-ca\") pod \"1a997e50-2b16-4fab-8556-138de2d1cc11\" (UID: \"1a997e50-2b16-4fab-8556-138de2d1cc11\") "
Oct 30 00:50:15.232796 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:15.232781    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a997e50-2b16-4fab-8556-138de2d1cc11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a997e50-2b16-4fab-8556-138de2d1cc11-maas-api-openshift-ca\") pod \"1a997e50-2b16-4fab-8556-138de2d1cc11\" (UID: \"1a997e50-2b16-4fab-8556-138de2d1cc11\") "
Oct 30 00:50:15.233134 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:15.233108    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a997e50-2b16-4fab-8556-138de2d1cc11-maas-api-openshift-ca" pod "1a997e50-2b16-4fab-8556-138de2d1cc11" (UID: "1a997e50-2b16-4fab-8556-138de2d1cc11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:15.233256 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:15.233243    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a997e50-2b16-4fab-8556-138de2d1cc11-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:15.234722 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:15.234677    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a997e50-2b16-4fab-8556-138de2d1cc11-maas-api-xks-ca" pod "1a997e50-2b16-4fab-8556-138de2d1cc11" (UID: "1a997e50-2b16-4fab-8556-138de2d1cc11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:50:15.234722 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:15.234680    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a997e50-2b16-4fab-8556-138de2d1cc11-kube-api-access-s6rvg" pod "1a997e50-2b16-4fab-8556-138de2d1cc11" (UID: "1a997e50-2b16-4fab-8556-138de2d1cc11"). InnerVolumeSpecName "kube-api-access-s6rvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:15.335104 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:15.335073    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a997e50-2b16-4fab-8556-138de2d1cc11-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:15.335104 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:15.335099    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6rvg\" (UniqueName: \"kubernetes.io/projected/1a997e50-2b16-4fab-8556-138de2d1cc11-kube-api-access-s6rvg\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:16.003177 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:16.003146    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s"
Oct 30 00:50:16.003372 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:16.003178    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s" event={"ID":"1a997e50-2b16-4fab-8556-138de2d1cc11","Type":"ContainerDied","Data":"f81868553fdae6c280809445282e5c38ad4e7dcbc3070a9219343edd029052a0"}
Oct 30 00:50:16.003372 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:16.003227    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f81868553fdae6c280809445282e5c38ad4e7dcbc3070a9219343edd029052a0"
Oct 30 00:50:16.629696 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:50:16.629655    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:50:18.838814 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.838735    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"]
Oct 30 00:50:18.839377 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.839359    6099 cpu_manager.go:414] "containerMap: removing container" podUID="1a997e50-2b16-4fab-8556-138de2d1cc11" containerName="revoke-keys"
Oct 30 00:50:18.839377 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.839377    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a997e50-2b16-4fab-8556-138de2d1cc11" containerName="revoke-keys"
Oct 30 00:50:18.839602 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.839503    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a997e50-2b16-4fab-8556-138de2d1cc11" containerName="revoke-keys"
Oct 30 00:50:18.845316 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.845294    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:18.868867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.868829    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"]
Oct 30 00:50:18.876557 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.876529    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"]
Oct 30 00:50:18.885836 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.885808    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:18.929649 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.929617    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"]
Oct 30 00:50:18.981763 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.981692    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:18.981943 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.981876    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:18.981943 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.981917    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:18.981943 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.981939    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:18.982108 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.982016    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82fp6\" (UniqueName: \"kubernetes.io/projected/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-kube-api-access-82fp6\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:18.982108 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.982077    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:18.982185 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.982105    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:18.982219 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.982187    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:18.982219 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.982214    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:18.982311 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.982286    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59323d4-d59d-421d-8c1e-00570828759a-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:18.982364 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.982350    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79q4v\" (UniqueName: \"kubernetes.io/projected/c59323d4-d59d-421d-8c1e-00570828759a-kube-api-access-79q4v\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:18.982411 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.982375    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:18.982411 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.982393    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59323d4-d59d-421d-8c1e-00570828759a-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:18.982475 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.982468    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:18.982514 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.982496    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:18.982550 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.982522    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:18.982605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.982584    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59323d4-d59d-421d-8c1e-00570828759a-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:18.982646 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:18.982621    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.083701 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.083656    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.083701 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.083699    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.083983 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.083740    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.083983 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.083763    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82fp6\" (UniqueName: \"kubernetes.io/projected/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-kube-api-access-82fp6\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.083983 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.083792    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.083983 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.083814    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.083983 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.083847    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.083983 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.083885    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.083983 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.083940    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59323d4-d59d-421d-8c1e-00570828759a-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.083983 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.083967    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79q4v\" (UniqueName: \"kubernetes.io/projected/c59323d4-d59d-421d-8c1e-00570828759a-kube-api-access-79q4v\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.084393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.083997    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.084393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.084027    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59323d4-d59d-421d-8c1e-00570828759a-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.084393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.084095    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.084393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.084130    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.084393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.084142    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.084393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.084207    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.084393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.084219    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.084393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.084269    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59323d4-d59d-421d-8c1e-00570828759a-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.084393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.084301    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.084901 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.084403    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.084901 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.084423    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.084901 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.084541    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.084901 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.084812    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.084901 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.084864    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-credential-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.085194 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.085107    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-workload-socket\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.085262 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.085215    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-data\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.085586 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.085563    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-workload-certs\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.085980 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.085956    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59323d4-d59d-421d-8c1e-00570828759a-istiod-ca-cert\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.086607 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.086581    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.086751 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.086581    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-istio-envoy\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.087097 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.087078    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.087379 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.087361    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59323d4-d59d-421d-8c1e-00570828759a-istio-podinfo\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.105276 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.105216    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82fp6\" (UniqueName: \"kubernetes.io/projected/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-kube-api-access-82fp6\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.106134 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.106111    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59323d4-d59d-421d-8c1e-00570828759a-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.106252 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.106230    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79q4v\" (UniqueName: \"kubernetes.io/projected/c59323d4-d59d-421d-8c1e-00570828759a-kube-api-access-79q4v\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.106365 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.106338    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-token\") pod \"e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") " pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.164382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.164351    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:19.226743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.226563    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:19.517146 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.517110    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"]
Oct 30 00:50:19.534842 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.534077    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:50:19.534842 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.534167    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:50:19.534842 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.534214    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:50:19.634983 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.628511    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"]
Oct 30 00:50:19.651599 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:50:19.647218    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad32816c_7a8e_47c7_9f1c_3dfea9f917ad.slice/crio-3dcc3dc507d06ade74bbc6a91f6747f224010ca9de8062e9704ae19fde03c1ed WatchSource:0}: Error finding container 3dcc3dc507d06ade74bbc6a91f6747f224010ca9de8062e9704ae19fde03c1ed: Status 404 returned error can't find the container with id 3dcc3dc507d06ade74bbc6a91f6747f224010ca9de8062e9704ae19fde03c1ed
Oct 30 00:50:19.651905 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.651835    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:50:19.651966 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.651925    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:50:19.652029 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:19.651972    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:50:20.133741 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:20.126654    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7" event={"ID":"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad","Type":"ContainerStarted","Data":"666d6027f515804e409c376da07601c3d50231167fd1fd38e07e0847473e91d0"}
Oct 30 00:50:20.146860 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:20.146823    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7" event={"ID":"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad","Type":"ContainerStarted","Data":"3dcc3dc507d06ade74bbc6a91f6747f224010ca9de8062e9704ae19fde03c1ed"}
Oct 30 00:50:20.164745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:20.155799    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml" event={"ID":"c59323d4-d59d-421d-8c1e-00570828759a","Type":"ContainerStarted","Data":"8738655a94f271c11a6fa731543fafe2787d2f55dad36485cfedc11ef9204583"}
Oct 30 00:50:20.164745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:20.155853    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml" event={"ID":"c59323d4-d59d-421d-8c1e-00570828759a","Type":"ContainerStarted","Data":"417ce9dc8bb695e075c927ae1dc3e3bd740ad9132f3768b673a0e3500a0d9f66"}
Oct 30 00:50:20.171244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:20.170778    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:20.171244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:20.170909    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:50:20.171244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:20.170956    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml" podUID="c59323d4-d59d-421d-8c1e-00570828759a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 00:50:20.217730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:20.217151    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7" podStartSLOduration=2.21713093 podStartE2EDuration="2.21713093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:20.203283265 +0000 UTC m=+914.060191021" watchObservedRunningTime="2026-10-30 00:50:20.21713093 +0000 UTC m=+914.074038686"
Oct 30 00:50:20.241730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:20.233038    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:20.260746 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:20.258794    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 00:50:20.260746 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:20.258849    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7" podUID="ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 00:50:20.357662 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:20.357596    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml" podStartSLOduration=2.3575753329999998 podStartE2EDuration="2.357575333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:20.256368074 +0000 UTC m=+914.113275829" watchObservedRunningTime="2026-10-30 00:50:20.357575333 +0000 UTC m=+914.214483090"
Oct 30 00:50:21.193732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:21.193156    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:50:21.193732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:21.193217    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml" podUID="c59323d4-d59d-421d-8c1e-00570828759a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 00:50:21.230730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:21.230131    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 00:50:21.230730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:21.230192    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7" podUID="ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 00:50:21.278732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:21.278042    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:50:21.330733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:21.329799    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:21.342731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:21.340794    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:21.483790 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:21.482821    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:21.689730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:21.680822    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:21.703767 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:21.693815    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:21.919731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:21.918834    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:22.021731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:22.020811    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:22.170854 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:22.170517    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:50:22.170854 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:22.170606    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml" podUID="c59323d4-d59d-421d-8c1e-00570828759a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 00:50:22.231736 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:22.229075    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 00:50:22.231736 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:22.229146    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7" podUID="ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 00:50:22.288731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:22.284802    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:50:22.338730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:22.330809    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:22.364221 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:22.364174    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:22.416914 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:22.399970    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:50:22.417849 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:22.417821    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:50:22.418076 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:22.418060    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:50:22.538690 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:22.538611    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:50:22.732975 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:22.732941    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:50:22.762726 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:22.743488    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:50:23.214750 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:23.177125    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:50:23.214750 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:23.214202    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml" podUID="c59323d4-d59d-421d-8c1e-00570828759a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 00:50:23.227254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:23.227224    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 00:50:23.227403 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:23.227274    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7" podUID="ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 00:50:23.852197 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:23.852156    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58"]
Oct 30 00:50:23.904978 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:23.904942    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58"]
Oct 30 00:50:23.905344 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:23.905324    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58"
Oct 30 00:50:23.941320 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:23.940471    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:50:23.950820 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:23.950788    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:50:23.950953 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:23.950928    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:50:23.951056 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:23.950805    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:50:23.962361 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:23.962332    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b652b50a-c0b1-423a-af82-5cc0357ec187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-828c35b8-k9s58\" (UID: \"b652b50a-c0b1-423a-af82-5cc0357ec187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58"
Oct 30 00:50:23.962618 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:23.962598    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2997t\" (UniqueName: \"kubernetes.io/projected/b652b50a-c0b1-423a-af82-5cc0357ec187-kube-api-access-2997t\") pod \"maas-api-revoke-sub-828c35b8-k9s58\" (UID: \"b652b50a-c0b1-423a-af82-5cc0357ec187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58"
Oct 30 00:50:23.962898 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:23.962877    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b652b50a-c0b1-423a-af82-5cc0357ec187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-828c35b8-k9s58\" (UID: \"b652b50a-c0b1-423a-af82-5cc0357ec187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58"
Oct 30 00:50:24.064289 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:24.064253    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2997t\" (UniqueName: \"kubernetes.io/projected/b652b50a-c0b1-423a-af82-5cc0357ec187-kube-api-access-2997t\") pod \"maas-api-revoke-sub-828c35b8-k9s58\" (UID: \"b652b50a-c0b1-423a-af82-5cc0357ec187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58"
Oct 30 00:50:24.064461 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:24.064439    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b652b50a-c0b1-423a-af82-5cc0357ec187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-828c35b8-k9s58\" (UID: \"b652b50a-c0b1-423a-af82-5cc0357ec187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58"
Oct 30 00:50:24.064551 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:24.064479    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b652b50a-c0b1-423a-af82-5cc0357ec187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-828c35b8-k9s58\" (UID: \"b652b50a-c0b1-423a-af82-5cc0357ec187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58"
Oct 30 00:50:24.079733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:24.065311    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b652b50a-c0b1-423a-af82-5cc0357ec187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-828c35b8-k9s58\" (UID: \"b652b50a-c0b1-423a-af82-5cc0357ec187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58"
Oct 30 00:50:24.120032 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:24.119897    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b652b50a-c0b1-423a-af82-5cc0357ec187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-828c35b8-k9s58\" (UID: \"b652b50a-c0b1-423a-af82-5cc0357ec187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58"
Oct 30 00:50:24.124642 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:24.124591    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2997t\" (UniqueName: \"kubernetes.io/projected/b652b50a-c0b1-423a-af82-5cc0357ec187-kube-api-access-2997t\") pod \"maas-api-revoke-sub-828c35b8-k9s58\" (UID: \"b652b50a-c0b1-423a-af82-5cc0357ec187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58"
Oct 30 00:50:24.187283 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:24.187247    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:50:24.187546 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:24.187517    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml" podUID="c59323d4-d59d-421d-8c1e-00570828759a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 00:50:24.227982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:24.227936    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 00:50:24.228231 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:24.228211    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7" podUID="ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 00:50:24.253256 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:24.253226    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58"
Oct 30 00:50:24.728325 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:24.728269    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58"]
Oct 30 00:50:25.185754 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:25.175754    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:50:25.185754 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:25.175805    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml" podUID="c59323d4-d59d-421d-8c1e-00570828759a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 00:50:25.227685 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:25.227653    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 00:50:25.227797 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:25.227696    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7" podUID="ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 00:50:25.343214 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:25.343179    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58" event={"ID":"b652b50a-c0b1-423a-af82-5cc0357ec187","Type":"ContainerStarted","Data":"4dee8ac89c37983f44b795560a9bf0599710b26a966f85235e6ed6df89ac4cdc"}
Oct 30 00:50:25.343397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:25.343233    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58" event={"ID":"b652b50a-c0b1-423a-af82-5cc0357ec187","Type":"ContainerStarted","Data":"9278d5f4fe7df7b5d3317b92ffd84f1eaf1ba94c63ad5b894611fa01f43cb7cd"}
Oct 30 00:50:26.169738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:26.169099    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:50:26.169738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:26.169165    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml" podUID="c59323d4-d59d-421d-8c1e-00570828759a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 00:50:26.227268 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:26.227193    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 00:50:26.227268 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:26.227232    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7" podUID="ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 00:50:26.352216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:26.352182    6099 generic.go:356] "Generic (PLEG): container finished" podID="b652b50a-c0b1-423a-af82-5cc0357ec187" containerID="4dee8ac89c37983f44b795560a9bf0599710b26a966f85235e6ed6df89ac4cdc" exitCode=0
Oct 30 00:50:26.352374 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:26.352246    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58" event={"ID":"b652b50a-c0b1-423a-af82-5cc0357ec187","Type":"ContainerDied","Data":"4dee8ac89c37983f44b795560a9bf0599710b26a966f85235e6ed6df89ac4cdc"}
Oct 30 00:50:27.165467 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.165398    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 00:50:27.165662 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.165494    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml" podUID="c59323d4-d59d-421d-8c1e-00570828759a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 00:50:27.185479 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.185442    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"]
Oct 30 00:50:27.191069 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.191040    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.215884 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.215860    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"]
Oct 30 00:50:27.220674 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.220645    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5062f6\"/\"distinct-w6-5062f6-kserve-self-signed-certs\""
Oct 30 00:50:27.227436 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.227403    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 00:50:27.227888 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.227457    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7" podUID="ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 00:50:27.275372 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.275329    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-model-cache\") pod \"distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.275550 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.275526    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-tmp-dir\") pod \"distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.275614 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.275587    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf04d49-5a22-4e28-b566-a3cb4306ad08-tls-certs\") pod \"distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.275721 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.275686    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5ffl\" (UniqueName: \"kubernetes.io/projected/dcf04d49-5a22-4e28-b566-a3cb4306ad08-kube-api-access-k5ffl\") pod \"distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.275877 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.275855    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-home\") pod \"distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.275954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.275897    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-dshm\") pod \"distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.377061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.377029    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-tmp-dir\") pod \"distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.377224 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.377197    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf04d49-5a22-4e28-b566-a3cb4306ad08-tls-certs\") pod \"distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.377291 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.377275    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5ffl\" (UniqueName: \"kubernetes.io/projected/dcf04d49-5a22-4e28-b566-a3cb4306ad08-kube-api-access-k5ffl\") pod \"distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.377414 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.377393    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-home\") pod \"distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.377478 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.377440    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-dshm\") pod \"distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.377478 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.377461    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-tmp-dir\") pod \"distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.377643 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.377621    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-model-cache\") pod \"distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.377943 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.377916    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-model-cache\") pod \"distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.378274 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.378245    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-home\") pod \"distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.380046 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.380024    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-dshm\") pod \"distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.380483 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.380466    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf04d49-5a22-4e28-b566-a3cb4306ad08-tls-certs\") pod \"distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.402584 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.402549    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5ffl\" (UniqueName: \"kubernetes.io/projected/dcf04d49-5a22-4e28-b566-a3cb4306ad08-kube-api-access-k5ffl\") pod \"distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.526343 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.510389    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:27.670756 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.670575    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58"
Oct 30 00:50:27.708646 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.708619    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b652b50a-c0b1-423a-af82-5cc0357ec187-kube-api-access-2997t\" (UniqueName: \"kubernetes.io/projected/b652b50a-c0b1-423a-af82-5cc0357ec187-kube-api-access-2997t\") pod \"b652b50a-c0b1-423a-af82-5cc0357ec187\" (UID: \"b652b50a-c0b1-423a-af82-5cc0357ec187\") "
Oct 30 00:50:27.708905 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.708671    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b652b50a-c0b1-423a-af82-5cc0357ec187-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b652b50a-c0b1-423a-af82-5cc0357ec187-maas-api-xks-ca\") pod \"b652b50a-c0b1-423a-af82-5cc0357ec187\" (UID: \"b652b50a-c0b1-423a-af82-5cc0357ec187\") "
Oct 30 00:50:27.708905 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.708786    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b652b50a-c0b1-423a-af82-5cc0357ec187-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b652b50a-c0b1-423a-af82-5cc0357ec187-maas-api-openshift-ca\") pod \"b652b50a-c0b1-423a-af82-5cc0357ec187\" (UID: \"b652b50a-c0b1-423a-af82-5cc0357ec187\") "
Oct 30 00:50:27.724738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.724106    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b652b50a-c0b1-423a-af82-5cc0357ec187-maas-api-openshift-ca" pod "b652b50a-c0b1-423a-af82-5cc0357ec187" (UID: "b652b50a-c0b1-423a-af82-5cc0357ec187"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:27.726414 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.726367    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b652b50a-c0b1-423a-af82-5cc0357ec187-maas-api-xks-ca" pod "b652b50a-c0b1-423a-af82-5cc0357ec187" (UID: "b652b50a-c0b1-423a-af82-5cc0357ec187"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:50:27.729357 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.729328    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b652b50a-c0b1-423a-af82-5cc0357ec187-kube-api-access-2997t" pod "b652b50a-c0b1-423a-af82-5cc0357ec187" (UID: "b652b50a-c0b1-423a-af82-5cc0357ec187"). InnerVolumeSpecName "kube-api-access-2997t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:27.812853 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.812816    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b652b50a-c0b1-423a-af82-5cc0357ec187-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:27.812853 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.812844    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2997t\" (UniqueName: \"kubernetes.io/projected/b652b50a-c0b1-423a-af82-5cc0357ec187-kube-api-access-2997t\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:27.812853 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.812853    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b652b50a-c0b1-423a-af82-5cc0357ec187-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:27.877288 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:27.877254    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"]
Oct 30 00:50:27.882147 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:50:27.882120    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcf04d49_5a22_4e28_b566_a3cb4306ad08.slice/crio-d9c6358d24164764053764e331391b7c1cf26727d7909fba96245fd4abcdd966 WatchSource:0}: Error finding container d9c6358d24164764053764e331391b7c1cf26727d7909fba96245fd4abcdd966: Status 404 returned error can't find the container with id d9c6358d24164764053764e331391b7c1cf26727d7909fba96245fd4abcdd966
Oct 30 00:50:28.374423 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:28.374386    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58"
Oct 30 00:50:28.375216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:28.375190    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58" event={"ID":"b652b50a-c0b1-423a-af82-5cc0357ec187","Type":"ContainerDied","Data":"9278d5f4fe7df7b5d3317b92ffd84f1eaf1ba94c63ad5b894611fa01f43cb7cd"}
Oct 30 00:50:28.375356 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:28.375243    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9278d5f4fe7df7b5d3317b92ffd84f1eaf1ba94c63ad5b894611fa01f43cb7cd"
Oct 30 00:50:28.378088 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:28.378060    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" event={"ID":"dcf04d49-5a22-4e28-b566-a3cb4306ad08","Type":"ContainerStarted","Data":"3a980ce7dee14d9664cf0b8ecf5377a3ea3f8c7b7204da49d9126599e400b2fe"}
Oct 30 00:50:28.378208 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:28.378101    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" event={"ID":"dcf04d49-5a22-4e28-b566-a3cb4306ad08","Type":"ContainerStarted","Data":"d9c6358d24164764053764e331391b7c1cf26727d7909fba96245fd4abcdd966"}
Oct 30 00:50:28.431594 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:28.431523    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podStartSLOduration=1.431505585 podStartE2EDuration="1.431505585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:28.41752788 +0000 UTC m=+922.274435636" watchObservedRunningTime="2026-10-30 00:50:28.431505585 +0000 UTC m=+922.288413340"
Oct 30 00:50:28.514874 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:28.514832    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:28.608789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:28.608758    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:28.609678 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:28.609610    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:28.610302 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:28.610282    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:50:28.672032 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:28.671992    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:28.672411 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:28.672391    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:28.673156 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:28.673135    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:50:33.856372 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.856326    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"]
Oct 30 00:50:33.857471 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.857441    6099 cpu_manager.go:414] "containerMap: removing container" podUID="b652b50a-c0b1-423a-af82-5cc0357ec187" containerName="revoke-keys"
Oct 30 00:50:33.857471 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.857471    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b652b50a-c0b1-423a-af82-5cc0357ec187" containerName="revoke-keys"
Oct 30 00:50:33.857667 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.857651    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="b652b50a-c0b1-423a-af82-5cc0357ec187" containerName="revoke-keys"
Oct 30 00:50:33.862205 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.862182    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:33.880675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.880647    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"]
Oct 30 00:50:33.887138 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.887113    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-40ac41\"/\"distinct-w2-40ac41-kserve-self-signed-certs\""
Oct 30 00:50:33.898079 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.898052    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-model-cache\") pod \"distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:33.898170 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.898134    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-dshm\") pod \"distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:33.898257 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.898225    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-home\") pod \"distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:33.898304 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.898268    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8279\" (UniqueName: \"kubernetes.io/projected/5daab803-f655-41c9-98d9-7049b1ba3597-kube-api-access-f8279\") pod \"distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:33.898351 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.898329    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5daab803-f655-41c9-98d9-7049b1ba3597-tls-certs\") pod \"distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:33.898459 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.898448    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-tmp-dir\") pod \"distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:33.999511 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.999477    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-dshm\") pod \"distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:33.999511 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.999510    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-home\") pod \"distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:33.999756 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.999528    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8279\" (UniqueName: \"kubernetes.io/projected/5daab803-f655-41c9-98d9-7049b1ba3597-kube-api-access-f8279\") pod \"distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:33.999756 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.999632    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5daab803-f655-41c9-98d9-7049b1ba3597-tls-certs\") pod \"distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:33.999860 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.999775    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-tmp-dir\") pod \"distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:33.999942 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.999915    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-home\") pod \"distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:34.000025 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:33.999922    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-model-cache\") pod \"distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:34.000143 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:34.000114    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-tmp-dir\") pod \"distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:34.000292 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:34.000179    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-model-cache\") pod \"distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:34.001813 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:34.001792    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-dshm\") pod \"distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:34.002182 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:34.002164    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5daab803-f655-41c9-98d9-7049b1ba3597-tls-certs\") pod \"distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:34.029402 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:34.029374    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8279\" (UniqueName: \"kubernetes.io/projected/5daab803-f655-41c9-98d9-7049b1ba3597-kube-api-access-f8279\") pod \"distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") " pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:34.182639 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:34.182538    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:34.504888 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:34.504858    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"]
Oct 30 00:50:34.505854 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:50:34.505824    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5daab803_f655_41c9_98d9_7049b1ba3597.slice/crio-3cb9ba7639d608e779d36a1496acd498a56664b82f6726c9197ee3b912b947c4 WatchSource:0}: Error finding container 3cb9ba7639d608e779d36a1496acd498a56664b82f6726c9197ee3b912b947c4: Status 404 returned error can't find the container with id 3cb9ba7639d608e779d36a1496acd498a56664b82f6726c9197ee3b912b947c4
Oct 30 00:50:35.437205 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:35.437165    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" event={"ID":"5daab803-f655-41c9-98d9-7049b1ba3597","Type":"ContainerStarted","Data":"fcbefd856a76c823e0f9502b98fabf035e638089a38641cc1fbd6ba7da4f02e8"}
Oct 30 00:50:35.437639 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:35.437217    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" event={"ID":"5daab803-f655-41c9-98d9-7049b1ba3597","Type":"ContainerStarted","Data":"3cb9ba7639d608e779d36a1496acd498a56664b82f6726c9197ee3b912b947c4"}
Oct 30 00:50:35.480089 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:35.480020    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podStartSLOduration=2.480002473 podStartE2EDuration="2.480002473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:35.479224572 +0000 UTC m=+929.336132327" watchObservedRunningTime="2026-10-30 00:50:35.480002473 +0000 UTC m=+929.336910230"
Oct 30 00:50:36.183767 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:36.183700    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:37.514232 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:37.514193    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:37.526220 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:37.526194    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:38.470410 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:38.470376    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:38.924906 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:38.924871    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls"]
Oct 30 00:50:38.929830 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:38.929812    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls"
Oct 30 00:50:38.944516 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:38.944493    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:50:38.944620 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:38.944554    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:50:38.944620 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:38.944614    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:50:38.944773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:38.944762    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:50:38.947366 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:38.947345    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls"]
Oct 30 00:50:39.072288 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:39.072256    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9fa6568-b070-4871-8188-6cec656f3094-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-800799d3-h6dls\" (UID: \"d9fa6568-b070-4871-8188-6cec656f3094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls"
Oct 30 00:50:39.072451 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:39.072413    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9skd\" (UniqueName: \"kubernetes.io/projected/d9fa6568-b070-4871-8188-6cec656f3094-kube-api-access-m9skd\") pod \"maas-api-revoke-sub-800799d3-h6dls\" (UID: \"d9fa6568-b070-4871-8188-6cec656f3094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls"
Oct 30 00:50:39.072528 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:39.072512    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9fa6568-b070-4871-8188-6cec656f3094-maas-api-xks-ca\") pod \"maas-api-revoke-sub-800799d3-h6dls\" (UID: \"d9fa6568-b070-4871-8188-6cec656f3094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls"
Oct 30 00:50:39.173637 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:39.173608    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9skd\" (UniqueName: \"kubernetes.io/projected/d9fa6568-b070-4871-8188-6cec656f3094-kube-api-access-m9skd\") pod \"maas-api-revoke-sub-800799d3-h6dls\" (UID: \"d9fa6568-b070-4871-8188-6cec656f3094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls"
Oct 30 00:50:39.173786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:39.173698    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9fa6568-b070-4871-8188-6cec656f3094-maas-api-xks-ca\") pod \"maas-api-revoke-sub-800799d3-h6dls\" (UID: \"d9fa6568-b070-4871-8188-6cec656f3094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls"
Oct 30 00:50:39.173786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:39.173776    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9fa6568-b070-4871-8188-6cec656f3094-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-800799d3-h6dls\" (UID: \"d9fa6568-b070-4871-8188-6cec656f3094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls"
Oct 30 00:50:39.174397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:39.174374    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9fa6568-b070-4871-8188-6cec656f3094-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-800799d3-h6dls\" (UID: \"d9fa6568-b070-4871-8188-6cec656f3094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls"
Oct 30 00:50:39.176128 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:39.176075    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9fa6568-b070-4871-8188-6cec656f3094-maas-api-xks-ca\") pod \"maas-api-revoke-sub-800799d3-h6dls\" (UID: \"d9fa6568-b070-4871-8188-6cec656f3094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls"
Oct 30 00:50:39.196427 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:39.196405    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9skd\" (UniqueName: \"kubernetes.io/projected/d9fa6568-b070-4871-8188-6cec656f3094-kube-api-access-m9skd\") pod \"maas-api-revoke-sub-800799d3-h6dls\" (UID: \"d9fa6568-b070-4871-8188-6cec656f3094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls"
Oct 30 00:50:39.244492 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:39.244458    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls"
Oct 30 00:50:39.544157 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:50:39.544132    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9fa6568_b070_4871_8188_6cec656f3094.slice/crio-c65ab184574bce59afa745eccf0d1b2806c8ad0aa5e5a49bc97999038fbf5de7 WatchSource:0}: Error finding container c65ab184574bce59afa745eccf0d1b2806c8ad0aa5e5a49bc97999038fbf5de7: Status 404 returned error can't find the container with id c65ab184574bce59afa745eccf0d1b2806c8ad0aa5e5a49bc97999038fbf5de7
Oct 30 00:50:39.546700 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:39.546672    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls"]
Oct 30 00:50:40.481456 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:40.481415    6099 generic.go:356] "Generic (PLEG): container finished" podID="d9fa6568-b070-4871-8188-6cec656f3094" containerID="b3bc3a4dafc51fe6434186b8a79191a33c26736ad36ea1201968debfff73961a" exitCode=0
Oct 30 00:50:40.482002 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:40.481641    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls" event={"ID":"d9fa6568-b070-4871-8188-6cec656f3094","Type":"ContainerDied","Data":"b3bc3a4dafc51fe6434186b8a79191a33c26736ad36ea1201968debfff73961a"}
Oct 30 00:50:40.482002 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:40.481690    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls" event={"ID":"d9fa6568-b070-4871-8188-6cec656f3094","Type":"ContainerStarted","Data":"c65ab184574bce59afa745eccf0d1b2806c8ad0aa5e5a49bc97999038fbf5de7"}
Oct 30 00:50:40.847453 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:40.847419    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9"]
Oct 30 00:50:40.852531 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:40.852506    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9"
Oct 30 00:50:40.869994 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:40.869967    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9"]
Oct 30 00:50:40.996442 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:40.996409    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3185456e-bwff9\" (UID: \"7c877c1a-9a08-4cf6-a881-b8f19ca4433b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9"
Oct 30 00:50:40.996577 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:40.996466    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3185456e-bwff9\" (UID: \"7c877c1a-9a08-4cf6-a881-b8f19ca4433b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9"
Oct 30 00:50:40.996622 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:40.996608    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp6kc\" (UniqueName: \"kubernetes.io/projected/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-kube-api-access-mp6kc\") pod \"maas-api-revoke-sub-3185456e-bwff9\" (UID: \"7c877c1a-9a08-4cf6-a881-b8f19ca4433b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9"
Oct 30 00:50:41.098219 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.098125    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3185456e-bwff9\" (UID: \"7c877c1a-9a08-4cf6-a881-b8f19ca4433b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9"
Oct 30 00:50:41.098392 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.098238    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp6kc\" (UniqueName: \"kubernetes.io/projected/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-kube-api-access-mp6kc\") pod \"maas-api-revoke-sub-3185456e-bwff9\" (UID: \"7c877c1a-9a08-4cf6-a881-b8f19ca4433b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9"
Oct 30 00:50:41.098392 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.098365    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3185456e-bwff9\" (UID: \"7c877c1a-9a08-4cf6-a881-b8f19ca4433b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9"
Oct 30 00:50:41.098900 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.098874    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3185456e-bwff9\" (UID: \"7c877c1a-9a08-4cf6-a881-b8f19ca4433b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9"
Oct 30 00:50:41.100580 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.100562    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3185456e-bwff9\" (UID: \"7c877c1a-9a08-4cf6-a881-b8f19ca4433b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9"
Oct 30 00:50:41.121182 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.121160    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp6kc\" (UniqueName: \"kubernetes.io/projected/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-kube-api-access-mp6kc\") pod \"maas-api-revoke-sub-3185456e-bwff9\" (UID: \"7c877c1a-9a08-4cf6-a881-b8f19ca4433b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9"
Oct 30 00:50:41.166594 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.166559    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9"
Oct 30 00:50:41.469880 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.469847    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9"]
Oct 30 00:50:41.470121 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:50:41.470097    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c877c1a_9a08_4cf6_a881_b8f19ca4433b.slice/crio-f1c045281073fbeb702c6d19abcd41c2b024fd3c6cdf324e8121146c195f58f6 WatchSource:0}: Error finding container f1c045281073fbeb702c6d19abcd41c2b024fd3c6cdf324e8121146c195f58f6: Status 404 returned error can't find the container with id f1c045281073fbeb702c6d19abcd41c2b024fd3c6cdf324e8121146c195f58f6
Oct 30 00:50:41.491576 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.491547    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9" event={"ID":"7c877c1a-9a08-4cf6-a881-b8f19ca4433b","Type":"ContainerStarted","Data":"f1c045281073fbeb702c6d19abcd41c2b024fd3c6cdf324e8121146c195f58f6"}
Oct 30 00:50:41.628768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.628746    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls"
Oct 30 00:50:41.706753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.706700    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9fa6568-b070-4871-8188-6cec656f3094-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9fa6568-b070-4871-8188-6cec656f3094-maas-api-xks-ca\") pod \"d9fa6568-b070-4871-8188-6cec656f3094\" (UID: \"d9fa6568-b070-4871-8188-6cec656f3094\") "
Oct 30 00:50:41.706923 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.706904    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9fa6568-b070-4871-8188-6cec656f3094-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9fa6568-b070-4871-8188-6cec656f3094-maas-api-openshift-ca\") pod \"d9fa6568-b070-4871-8188-6cec656f3094\" (UID: \"d9fa6568-b070-4871-8188-6cec656f3094\") "
Oct 30 00:50:41.706983 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.706952    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9fa6568-b070-4871-8188-6cec656f3094-kube-api-access-m9skd\" (UniqueName: \"kubernetes.io/projected/d9fa6568-b070-4871-8188-6cec656f3094-kube-api-access-m9skd\") pod \"d9fa6568-b070-4871-8188-6cec656f3094\" (UID: \"d9fa6568-b070-4871-8188-6cec656f3094\") "
Oct 30 00:50:41.707301 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.707277    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9fa6568-b070-4871-8188-6cec656f3094-maas-api-openshift-ca" pod "d9fa6568-b070-4871-8188-6cec656f3094" (UID: "d9fa6568-b070-4871-8188-6cec656f3094"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:41.707664 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.707641    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9fa6568-b070-4871-8188-6cec656f3094-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:41.709487 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.709455    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9fa6568-b070-4871-8188-6cec656f3094-maas-api-xks-ca" pod "d9fa6568-b070-4871-8188-6cec656f3094" (UID: "d9fa6568-b070-4871-8188-6cec656f3094"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:50:41.709608 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.709589    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9fa6568-b070-4871-8188-6cec656f3094-kube-api-access-m9skd" pod "d9fa6568-b070-4871-8188-6cec656f3094" (UID: "d9fa6568-b070-4871-8188-6cec656f3094"). InnerVolumeSpecName "kube-api-access-m9skd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:41.808481 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.808454    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9fa6568-b070-4871-8188-6cec656f3094-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:41.808573 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:41.808485    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9skd\" (UniqueName: \"kubernetes.io/projected/d9fa6568-b070-4871-8188-6cec656f3094-kube-api-access-m9skd\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:42.500545 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:42.500469    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls"
Oct 30 00:50:42.500545 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:42.500503    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls" event={"ID":"d9fa6568-b070-4871-8188-6cec656f3094","Type":"ContainerDied","Data":"c65ab184574bce59afa745eccf0d1b2806c8ad0aa5e5a49bc97999038fbf5de7"}
Oct 30 00:50:42.501026 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:42.500582    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c65ab184574bce59afa745eccf0d1b2806c8ad0aa5e5a49bc97999038fbf5de7"
Oct 30 00:50:42.502285 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:42.502259    6099 generic.go:356] "Generic (PLEG): container finished" podID="7c877c1a-9a08-4cf6-a881-b8f19ca4433b" containerID="e9c1aa64b232d640149da1c49f092b722435cb8486a0e9a959d8be2782492699" exitCode=0
Oct 30 00:50:42.502393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:42.502291    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9" event={"ID":"7c877c1a-9a08-4cf6-a881-b8f19ca4433b","Type":"ContainerDied","Data":"e9c1aa64b232d640149da1c49f092b722435cb8486a0e9a959d8be2782492699"}
Oct 30 00:50:43.055982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:43.055949    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:50:43.694735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:43.694413    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9"
Oct 30 00:50:43.834870 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:43.834836    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-maas-api-openshift-ca\") pod \"7c877c1a-9a08-4cf6-a881-b8f19ca4433b\" (UID: \"7c877c1a-9a08-4cf6-a881-b8f19ca4433b\") "
Oct 30 00:50:43.835077 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:43.835047    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-maas-api-xks-ca\") pod \"7c877c1a-9a08-4cf6-a881-b8f19ca4433b\" (UID: \"7c877c1a-9a08-4cf6-a881-b8f19ca4433b\") "
Oct 30 00:50:43.835149 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:43.835086    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-kube-api-access-mp6kc\" (UniqueName: \"kubernetes.io/projected/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-kube-api-access-mp6kc\") pod \"7c877c1a-9a08-4cf6-a881-b8f19ca4433b\" (UID: \"7c877c1a-9a08-4cf6-a881-b8f19ca4433b\") "
Oct 30 00:50:43.835591 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:43.835564    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-maas-api-openshift-ca" pod "7c877c1a-9a08-4cf6-a881-b8f19ca4433b" (UID: "7c877c1a-9a08-4cf6-a881-b8f19ca4433b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:43.835947 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:43.835923    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:43.837869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:43.837841    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-kube-api-access-mp6kc" pod "7c877c1a-9a08-4cf6-a881-b8f19ca4433b" (UID: "7c877c1a-9a08-4cf6-a881-b8f19ca4433b"). InnerVolumeSpecName "kube-api-access-mp6kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:43.837996 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:43.837892    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-maas-api-xks-ca" pod "7c877c1a-9a08-4cf6-a881-b8f19ca4433b" (UID: "7c877c1a-9a08-4cf6-a881-b8f19ca4433b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:50:43.937276 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:43.937246    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:43.937276 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:43.937276    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp6kc\" (UniqueName: \"kubernetes.io/projected/7c877c1a-9a08-4cf6-a881-b8f19ca4433b-kube-api-access-mp6kc\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:44.185784 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:44.185649    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:44.197856 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:44.197828    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:44.565728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:44.559944    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9"
Oct 30 00:50:44.565728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:44.560019    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9" event={"ID":"7c877c1a-9a08-4cf6-a881-b8f19ca4433b","Type":"ContainerDied","Data":"f1c045281073fbeb702c6d19abcd41c2b024fd3c6cdf324e8121146c195f58f6"}
Oct 30 00:50:44.565728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:44.560108    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1c045281073fbeb702c6d19abcd41c2b024fd3c6cdf324e8121146c195f58f6"
Oct 30 00:50:44.602291 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:44.602260    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:46.439731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:46.439613    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 00:50:46.464838 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:46.464791    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:46.481838 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:46.481789    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:46.540856 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:46.540806    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:46.568735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:46.567798    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:46.693788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:46.676657    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 00:50:46.718731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:46.711827    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 00:50:46.760731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:46.757580    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:46.784168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:46.761142    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:46.962789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:46.958777    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:47.152031 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:50:47.151964    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:50:47.338786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.293932    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:47.535752 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.448743    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:47.916731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.912502    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:50:47.916731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.554390    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 00:50:47.939868 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.939836    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:47.947047 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.758519    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:47.952861 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.952836    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:50:47.976552 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.760397    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:47.976806 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.809129    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:47.976952 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.809272    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:50:47.977109 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.848081    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:47.977244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.749537    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 00:50:47.977378 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.749577    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:47.977497 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.761646    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 00:50:47.977743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.977715    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:50:48.013333 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.991792    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:50:48.013333 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.991860    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:50:48.013333 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.991899    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:50:48.013333 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.991935    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:50:48.013333 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.991967    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:50:48.013333 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.992001    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:48.013333 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:47.992037    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:50:48.143491 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.143463    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:50:48.384567 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.384537    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:50:48.386640 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.385112    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:50:48.386640 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.385165    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:50:48.386640 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.385197    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:50:48.386640 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.386034    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:50:48.388271 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.387743    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:50:48.401880 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.401849    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:50:48.408951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.408934    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:50:48.415805 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.409960    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:50:48.605524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.605492    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b"]
Oct 30 00:50:48.606441 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.606417    6099 cpu_manager.go:414] "containerMap: removing container" podUID="d9fa6568-b070-4871-8188-6cec656f3094" containerName="revoke-keys"
Oct 30 00:50:48.606527 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.606443    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9fa6568-b070-4871-8188-6cec656f3094" containerName="revoke-keys"
Oct 30 00:50:48.606527 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.606515    6099 cpu_manager.go:414] "containerMap: removing container" podUID="7c877c1a-9a08-4cf6-a881-b8f19ca4433b" containerName="revoke-keys"
Oct 30 00:50:48.606527 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.606523    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c877c1a-9a08-4cf6-a881-b8f19ca4433b" containerName="revoke-keys"
Oct 30 00:50:48.606693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.606679    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c877c1a-9a08-4cf6-a881-b8f19ca4433b" containerName="revoke-keys"
Oct 30 00:50:48.606753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.606701    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9fa6568-b070-4871-8188-6cec656f3094" containerName="revoke-keys"
Oct 30 00:50:48.611340 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.611306    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b"
Oct 30 00:50:48.626076 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.626035    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:50:48.626899 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.626794    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:50:48.627209 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.627090    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:50:48.627209 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.627181    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:50:48.628862 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.628840    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b"]
Oct 30 00:50:48.706241 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.706151    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"]
Oct 30 00:50:48.711575 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.711550    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:48.741512 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.740862    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"]
Oct 30 00:50:48.792680 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.792643    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e366629-56cd-4c9e-bb4f-e489885fae80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7149df7-jp96b\" (UID: \"5e366629-56cd-4c9e-bb4f-e489885fae80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b"
Oct 30 00:50:48.792866 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.792812    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e366629-56cd-4c9e-bb4f-e489885fae80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7149df7-jp96b\" (UID: \"5e366629-56cd-4c9e-bb4f-e489885fae80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b"
Oct 30 00:50:48.792960 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.792923    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf68g\" (UniqueName: \"kubernetes.io/projected/5e366629-56cd-4c9e-bb4f-e489885fae80-kube-api-access-jf68g\") pod \"maas-api-revoke-sub-a7149df7-jp96b\" (UID: \"5e366629-56cd-4c9e-bb4f-e489885fae80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b"
Oct 30 00:50:48.828136 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.828083    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"]
Oct 30 00:50:48.834568 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.834535    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:48.862034 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.862004    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"]
Oct 30 00:50:48.894798 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.894751    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:48.894798 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.894798    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf68g\" (UniqueName: \"kubernetes.io/projected/5e366629-56cd-4c9e-bb4f-e489885fae80-kube-api-access-jf68g\") pod \"maas-api-revoke-sub-a7149df7-jp96b\" (UID: \"5e366629-56cd-4c9e-bb4f-e489885fae80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b"
Oct 30 00:50:48.895094 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.894834    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:48.895094 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.894894    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:48.895094 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.894919    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:48.895094 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.894952    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:48.895094 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.895042    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px97b\" (UniqueName: \"kubernetes.io/projected/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-kube-api-access-px97b\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:48.895094 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.895062    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:48.895094 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.895090    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:48.895457 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.895112    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e366629-56cd-4c9e-bb4f-e489885fae80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7149df7-jp96b\" (UID: \"5e366629-56cd-4c9e-bb4f-e489885fae80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b"
Oct 30 00:50:48.895457 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.895141    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:48.895457 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.895170    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e366629-56cd-4c9e-bb4f-e489885fae80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7149df7-jp96b\" (UID: \"5e366629-56cd-4c9e-bb4f-e489885fae80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b"
Oct 30 00:50:48.896269 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.896237    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e366629-56cd-4c9e-bb4f-e489885fae80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7149df7-jp96b\" (UID: \"5e366629-56cd-4c9e-bb4f-e489885fae80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b"
Oct 30 00:50:48.898666 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.898643    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e366629-56cd-4c9e-bb4f-e489885fae80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7149df7-jp96b\" (UID: \"5e366629-56cd-4c9e-bb4f-e489885fae80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b"
Oct 30 00:50:48.927633 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.927601    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf68g\" (UniqueName: \"kubernetes.io/projected/5e366629-56cd-4c9e-bb4f-e489885fae80-kube-api-access-jf68g\") pod \"maas-api-revoke-sub-a7149df7-jp96b\" (UID: \"5e366629-56cd-4c9e-bb4f-e489885fae80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b"
Oct 30 00:50:48.929853 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:48.929818    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b"
Oct 30 00:50:49.001534 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.001478    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.001675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.001550    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.001675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.001614    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px97b\" (UniqueName: \"kubernetes.io/projected/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-kube-api-access-px97b\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.001675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.001651    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.001896 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.001722    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.002285 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.002252    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-credential-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.003043 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.002588    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.003193 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.003126    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-workload-certs\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.003193 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.003146    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.003362 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.003208    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.003362 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.003242    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbsls\" (UniqueName: \"kubernetes.io/projected/74025dac-eaf2-43b0-82d0-5470bd908c97-kube-api-access-vbsls\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.003362 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.003277    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.003362 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.003334    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.003691 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.003398    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.003691 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.003426    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.003691 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.003434    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-workload-socket\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.004004 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.003837    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.004004 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.003877    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.004004 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.003910    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.004004 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.003933    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74025dac-eaf2-43b0-82d0-5470bd908c97-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.004004 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.003985    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.007167 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.005753    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-data\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.007167 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.006825    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istiod-ca-cert\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.009571 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.007348    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-podinfo\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.011924 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.011880    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-envoy\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.031003 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.030948    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px97b\" (UniqueName: \"kubernetes.io/projected/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-kube-api-access-px97b\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.031407 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.031374    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-token\") pod \"e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") " pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.105336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.105295    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74025dac-eaf2-43b0-82d0-5470bd908c97-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.105530 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.105473    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.105530 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.105522    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.107982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.106247    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74025dac-eaf2-43b0-82d0-5470bd908c97-istiod-ca-cert\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.107982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.106514    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-workload-certs\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.107982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.106580    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.107982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.106680    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.107982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.106735    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbsls\" (UniqueName: \"kubernetes.io/projected/74025dac-eaf2-43b0-82d0-5470bd908c97-kube-api-access-vbsls\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.107982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.106890    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.107982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.106915    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.107982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.107039    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.107982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.107603    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-data\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.107982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.107870    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-credential-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.107982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.107947    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-workload-socket\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.109110 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.109082    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-podinfo\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.109786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.109765    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-envoy\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.134488 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.134455    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-token\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.134613 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.134585    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbsls\" (UniqueName: \"kubernetes.io/projected/74025dac-eaf2-43b0-82d0-5470bd908c97-kube-api-access-vbsls\") pod \"e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") " pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.169458 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.169418    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:49.251241 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.251170    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b"]
Oct 30 00:50:49.329072 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.329045    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:49.480609 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.480582    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"]
Oct 30 00:50:49.480916 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:50:49.480895    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74025dac_eaf2_43b0_82d0_5470bd908c97.slice/crio-dfcbdb7860d9c2506485735bd70e89e424048cff615c600aee724f0e3f20c152 WatchSource:0}: Error finding container dfcbdb7860d9c2506485735bd70e89e424048cff615c600aee724f0e3f20c152: Status 404 returned error can't find the container with id dfcbdb7860d9c2506485735bd70e89e424048cff615c600aee724f0e3f20c152
Oct 30 00:50:49.483605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.483571    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:50:49.483748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.483649    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:50:49.483748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.483693    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:50:49.640831 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.640803    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"]
Oct 30 00:50:49.667658 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:50:49.667627    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e60f3b_c78f_4de5_aefe_d4e8ee216a84.slice/crio-dcbe38894977efafac0c55bd2b055b0b3bb4078cf0408dc841f2999575a34e46 WatchSource:0}: Error finding container dcbe38894977efafac0c55bd2b055b0b3bb4078cf0408dc841f2999575a34e46: Status 404 returned error can't find the container with id dcbe38894977efafac0c55bd2b055b0b3bb4078cf0408dc841f2999575a34e46
Oct 30 00:50:49.671038 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.670996    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:50:49.671122 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.671095    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:50:49.671218 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.671194    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:50:49.932953 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.932914    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd" event={"ID":"96e60f3b-c78f-4de5-aefe-d4e8ee216a84","Type":"ContainerStarted","Data":"6cb675297151a0c203afdb61c8cb245c1899acd1801873373314aafe7d66d6e6"}
Oct 30 00:50:49.933116 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.932985    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd" event={"ID":"96e60f3b-c78f-4de5-aefe-d4e8ee216a84","Type":"ContainerStarted","Data":"dcbe38894977efafac0c55bd2b055b0b3bb4078cf0408dc841f2999575a34e46"}
Oct 30 00:50:49.934875 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.934845    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b" event={"ID":"5e366629-56cd-4c9e-bb4f-e489885fae80","Type":"ContainerStarted","Data":"9f884e8d88195a356f55278d662e70abc75f4e05a0c0074d58a07f7ca6532678"}
Oct 30 00:50:49.935009 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.934904    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b" event={"ID":"5e366629-56cd-4c9e-bb4f-e489885fae80","Type":"ContainerStarted","Data":"856f7aa11ef776dc5294a61e4cddb6bd1075edca4afad6d919ed8b7a566d980c"}
Oct 30 00:50:49.936805 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.936782    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp" event={"ID":"74025dac-eaf2-43b0-82d0-5470bd908c97","Type":"ContainerStarted","Data":"63354bb26bba83abd609bc2456a2f383061ae431923ea8dd348e91e375a0ed7c"}
Oct 30 00:50:49.936909 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.936825    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp" event={"ID":"74025dac-eaf2-43b0-82d0-5470bd908c97","Type":"ContainerStarted","Data":"dfcbdb7860d9c2506485735bd70e89e424048cff615c600aee724f0e3f20c152"}
Oct 30 00:50:49.984644 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:49.984568    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd" podStartSLOduration=1.984538878 podStartE2EDuration="1.984538878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:49.984197858 +0000 UTC m=+943.841105628" watchObservedRunningTime="2026-10-30 00:50:49.984538878 +0000 UTC m=+943.841446633"
Oct 30 00:50:50.031555 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:50.031442    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b" podStartSLOduration=2.031424601 podStartE2EDuration="2.031424601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:50.026441968 +0000 UTC m=+943.883349721" watchObservedRunningTime="2026-10-30 00:50:50.031424601 +0000 UTC m=+943.888332356"
Oct 30 00:50:50.074181 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:50.074130    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp" podStartSLOduration=2.074117365 podStartE2EDuration="2.074117365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:50.072777625 +0000 UTC m=+943.929685380" watchObservedRunningTime="2026-10-30 00:50:50.074117365 +0000 UTC m=+943.931025120"
Oct 30 00:50:50.169602 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:50.169574    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:50.170574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:50.170552    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 00:50:50.170691 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:50.170607    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp" podUID="74025dac-eaf2-43b0-82d0-5470bd908c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 00:50:50.329190 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:50.329163    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:50.330483 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:50.330459    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 00:50:50.330605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:50.330505    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd" podUID="96e60f3b-c78f-4de5-aefe-d4e8ee216a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 00:50:50.946168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:50.946135    6099 generic.go:356] "Generic (PLEG): container finished" podID="5e366629-56cd-4c9e-bb4f-e489885fae80" containerID="9f884e8d88195a356f55278d662e70abc75f4e05a0c0074d58a07f7ca6532678" exitCode=0
Oct 30 00:50:50.946636 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:50.946187    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b" event={"ID":"5e366629-56cd-4c9e-bb4f-e489885fae80","Type":"ContainerDied","Data":"9f884e8d88195a356f55278d662e70abc75f4e05a0c0074d58a07f7ca6532678"}
Oct 30 00:50:51.190732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:51.190059    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 00:50:51.190732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:51.190122    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp" podUID="74025dac-eaf2-43b0-82d0-5470bd908c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 00:50:51.329468 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:51.329434    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 00:50:51.329650 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:51.329484    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd" podUID="96e60f3b-c78f-4de5-aefe-d4e8ee216a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 00:50:52.222628 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:52.222596    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 00:50:52.223166 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:52.223126    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp" podUID="74025dac-eaf2-43b0-82d0-5470bd908c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 00:50:52.252395 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:52.252375    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b"
Oct 30 00:50:52.260291 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:52.260212    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e366629-56cd-4c9e-bb4f-e489885fae80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e366629-56cd-4c9e-bb4f-e489885fae80-maas-api-openshift-ca\") pod \"5e366629-56cd-4c9e-bb4f-e489885fae80\" (UID: \"5e366629-56cd-4c9e-bb4f-e489885fae80\") "
Oct 30 00:50:52.261265 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:52.260421    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e366629-56cd-4c9e-bb4f-e489885fae80-kube-api-access-jf68g\" (UniqueName: \"kubernetes.io/projected/5e366629-56cd-4c9e-bb4f-e489885fae80-kube-api-access-jf68g\") pod \"5e366629-56cd-4c9e-bb4f-e489885fae80\" (UID: \"5e366629-56cd-4c9e-bb4f-e489885fae80\") "
Oct 30 00:50:52.261265 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:52.260472    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e366629-56cd-4c9e-bb4f-e489885fae80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e366629-56cd-4c9e-bb4f-e489885fae80-maas-api-xks-ca\") pod \"5e366629-56cd-4c9e-bb4f-e489885fae80\" (UID: \"5e366629-56cd-4c9e-bb4f-e489885fae80\") "
Oct 30 00:50:52.261265 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:52.260528    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e366629-56cd-4c9e-bb4f-e489885fae80-maas-api-openshift-ca" pod "5e366629-56cd-4c9e-bb4f-e489885fae80" (UID: "5e366629-56cd-4c9e-bb4f-e489885fae80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:52.261500 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:52.261081    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e366629-56cd-4c9e-bb4f-e489885fae80-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:52.262662 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:52.262638    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e366629-56cd-4c9e-bb4f-e489885fae80-kube-api-access-jf68g" pod "5e366629-56cd-4c9e-bb4f-e489885fae80" (UID: "5e366629-56cd-4c9e-bb4f-e489885fae80"). InnerVolumeSpecName "kube-api-access-jf68g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:52.262958 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:52.262939    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e366629-56cd-4c9e-bb4f-e489885fae80-maas-api-xks-ca" pod "5e366629-56cd-4c9e-bb4f-e489885fae80" (UID: "5e366629-56cd-4c9e-bb4f-e489885fae80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:50:52.330131 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:52.329686    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 00:50:52.330131 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:52.329772    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd" podUID="96e60f3b-c78f-4de5-aefe-d4e8ee216a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 00:50:52.363804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:52.363081    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf68g\" (UniqueName: \"kubernetes.io/projected/5e366629-56cd-4c9e-bb4f-e489885fae80-kube-api-access-jf68g\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:52.363804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:52.363105    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e366629-56cd-4c9e-bb4f-e489885fae80-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:53.024317 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:53.024278    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b"
Oct 30 00:50:53.024517 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:53.024284    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b" event={"ID":"5e366629-56cd-4c9e-bb4f-e489885fae80","Type":"ContainerDied","Data":"856f7aa11ef776dc5294a61e4cddb6bd1075edca4afad6d919ed8b7a566d980c"}
Oct 30 00:50:53.024517 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:53.024437    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="856f7aa11ef776dc5294a61e4cddb6bd1075edca4afad6d919ed8b7a566d980c"
Oct 30 00:50:53.187732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:53.185234    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 00:50:53.187732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:53.185297    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp" podUID="74025dac-eaf2-43b0-82d0-5470bd908c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 00:50:53.329858 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:53.329823    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 00:50:53.330269 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:53.329869    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd" podUID="96e60f3b-c78f-4de5-aefe-d4e8ee216a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 00:50:54.016723 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.013752    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d"]
Oct 30 00:50:54.016723 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.014773    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5e366629-56cd-4c9e-bb4f-e489885fae80" containerName="revoke-keys"
Oct 30 00:50:54.016723 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.014795    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e366629-56cd-4c9e-bb4f-e489885fae80" containerName="revoke-keys"
Oct 30 00:50:54.016723 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.014944    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e366629-56cd-4c9e-bb4f-e489885fae80" containerName="revoke-keys"
Oct 30 00:50:54.025140 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.025115    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d"]
Oct 30 00:50:54.025449 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.025203    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d"
Oct 30 00:50:54.040614 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.040596    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:50:54.052588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.052569    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:50:54.052687 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.052671    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:50:54.052831 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.052814    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:50:54.138683 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.138651    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcqpt\" (UniqueName: \"kubernetes.io/projected/d0c42a5f-a350-4683-be19-4884131b1a42-kube-api-access-kcqpt\") pod \"maas-api-revoke-sub-17f56e0e-vnc2d\" (UID: \"d0c42a5f-a350-4683-be19-4884131b1a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d"
Oct 30 00:50:54.138884 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.138862    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c42a5f-a350-4683-be19-4884131b1a42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17f56e0e-vnc2d\" (UID: \"d0c42a5f-a350-4683-be19-4884131b1a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d"
Oct 30 00:50:54.138959 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.138898    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c42a5f-a350-4683-be19-4884131b1a42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17f56e0e-vnc2d\" (UID: \"d0c42a5f-a350-4683-be19-4884131b1a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d"
Oct 30 00:50:54.177731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.175060    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 00:50:54.177731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.175118    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp" podUID="74025dac-eaf2-43b0-82d0-5470bd908c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 00:50:54.240445 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.240411    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcqpt\" (UniqueName: \"kubernetes.io/projected/d0c42a5f-a350-4683-be19-4884131b1a42-kube-api-access-kcqpt\") pod \"maas-api-revoke-sub-17f56e0e-vnc2d\" (UID: \"d0c42a5f-a350-4683-be19-4884131b1a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d"
Oct 30 00:50:54.240674 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.240659    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c42a5f-a350-4683-be19-4884131b1a42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17f56e0e-vnc2d\" (UID: \"d0c42a5f-a350-4683-be19-4884131b1a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d"
Oct 30 00:50:54.240806 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.240777    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c42a5f-a350-4683-be19-4884131b1a42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17f56e0e-vnc2d\" (UID: \"d0c42a5f-a350-4683-be19-4884131b1a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d"
Oct 30 00:50:54.241448 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.241426    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c42a5f-a350-4683-be19-4884131b1a42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17f56e0e-vnc2d\" (UID: \"d0c42a5f-a350-4683-be19-4884131b1a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d"
Oct 30 00:50:54.244414 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.244377    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c42a5f-a350-4683-be19-4884131b1a42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17f56e0e-vnc2d\" (UID: \"d0c42a5f-a350-4683-be19-4884131b1a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d"
Oct 30 00:50:54.263250 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.263212    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcqpt\" (UniqueName: \"kubernetes.io/projected/d0c42a5f-a350-4683-be19-4884131b1a42-kube-api-access-kcqpt\") pod \"maas-api-revoke-sub-17f56e0e-vnc2d\" (UID: \"d0c42a5f-a350-4683-be19-4884131b1a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d"
Oct 30 00:50:54.329976 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.329948    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 00:50:54.330467 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.330439    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd" podUID="96e60f3b-c78f-4de5-aefe-d4e8ee216a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 00:50:54.342748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.342723    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d"
Oct 30 00:50:54.722257 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:54.722226    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d"]
Oct 30 00:50:54.722935 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:50:54.722904    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c42a5f_a350_4683_be19_4884131b1a42.slice/crio-0fe7cea2165b28cfcd855a7f1f568cf4ca559cb8f63641360bdad7c3f9e5d107 WatchSource:0}: Error finding container 0fe7cea2165b28cfcd855a7f1f568cf4ca559cb8f63641360bdad7c3f9e5d107: Status 404 returned error can't find the container with id 0fe7cea2165b28cfcd855a7f1f568cf4ca559cb8f63641360bdad7c3f9e5d107
Oct 30 00:50:55.047942 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:55.047899    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d" event={"ID":"d0c42a5f-a350-4683-be19-4884131b1a42","Type":"ContainerStarted","Data":"0fe7cea2165b28cfcd855a7f1f568cf4ca559cb8f63641360bdad7c3f9e5d107"}
Oct 30 00:50:55.175731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:55.174012    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 00:50:55.175731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:55.174069    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp" podUID="74025dac-eaf2-43b0-82d0-5470bd908c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 00:50:55.330289 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:55.330265    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 00:50:55.330600 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:55.330308    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd" podUID="96e60f3b-c78f-4de5-aefe-d4e8ee216a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 00:50:56.057468 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:56.057436    6099 generic.go:356] "Generic (PLEG): container finished" podID="d0c42a5f-a350-4683-be19-4884131b1a42" containerID="641cff1378937e8d9697f30b0a84dfe9cf0dfe8922058658e5d843686bead7dc" exitCode=0
Oct 30 00:50:56.057631 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:56.057489    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d" event={"ID":"d0c42a5f-a350-4683-be19-4884131b1a42","Type":"ContainerDied","Data":"641cff1378937e8d9697f30b0a84dfe9cf0dfe8922058658e5d843686bead7dc"}
Oct 30 00:50:56.193132 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:56.193085    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 00:50:56.193383 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:56.193347    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp" podUID="74025dac-eaf2-43b0-82d0-5470bd908c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 00:50:56.329943 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:56.329906    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 00:50:56.330126 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:56.329967    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd" podUID="96e60f3b-c78f-4de5-aefe-d4e8ee216a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 00:50:57.334061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:57.334027    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d"
Oct 30 00:50:57.414126 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:57.414037    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:57.416380 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:57.416228    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:57.417509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:57.417434    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 00:50:57.451789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:57.451754    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0c42a5f-a350-4683-be19-4884131b1a42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c42a5f-a350-4683-be19-4884131b1a42-maas-api-xks-ca\") pod \"d0c42a5f-a350-4683-be19-4884131b1a42\" (UID: \"d0c42a5f-a350-4683-be19-4884131b1a42\") "
Oct 30 00:50:57.452046 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:57.452023    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c42a5f-a350-4683-be19-4884131b1a42-kube-api-access-kcqpt\" (UniqueName: \"kubernetes.io/projected/d0c42a5f-a350-4683-be19-4884131b1a42-kube-api-access-kcqpt\") pod \"d0c42a5f-a350-4683-be19-4884131b1a42\" (UID: \"d0c42a5f-a350-4683-be19-4884131b1a42\") "
Oct 30 00:50:57.452160 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:57.452069    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c42a5f-a350-4683-be19-4884131b1a42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c42a5f-a350-4683-be19-4884131b1a42-maas-api-openshift-ca\") pod \"d0c42a5f-a350-4683-be19-4884131b1a42\" (UID: \"d0c42a5f-a350-4683-be19-4884131b1a42\") "
Oct 30 00:50:57.454290 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:57.454257    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c42a5f-a350-4683-be19-4884131b1a42-maas-api-openshift-ca" pod "d0c42a5f-a350-4683-be19-4884131b1a42" (UID: "d0c42a5f-a350-4683-be19-4884131b1a42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:57.454405 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:57.454319    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c42a5f-a350-4683-be19-4884131b1a42-maas-api-xks-ca" pod "d0c42a5f-a350-4683-be19-4884131b1a42" (UID: "d0c42a5f-a350-4683-be19-4884131b1a42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:50:57.456936 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:57.456905    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c42a5f-a350-4683-be19-4884131b1a42-kube-api-access-kcqpt" pod "d0c42a5f-a350-4683-be19-4884131b1a42" (UID: "d0c42a5f-a350-4683-be19-4884131b1a42"). InnerVolumeSpecName "kube-api-access-kcqpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:57.554419 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:57.554376    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c42a5f-a350-4683-be19-4884131b1a42-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:57.554419 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:57.554407    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcqpt\" (UniqueName: \"kubernetes.io/projected/d0c42a5f-a350-4683-be19-4884131b1a42-kube-api-access-kcqpt\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:57.554419 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:57.554417    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c42a5f-a350-4683-be19-4884131b1a42-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:50:57.708101 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:57.708020    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:57.709743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:57.708975    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:57.710579 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:57.710555    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:50:58.088680 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.088650    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d"
Oct 30 00:50:58.088883 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.088698    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d" event={"ID":"d0c42a5f-a350-4683-be19-4884131b1a42","Type":"ContainerDied","Data":"0fe7cea2165b28cfcd855a7f1f568cf4ca559cb8f63641360bdad7c3f9e5d107"}
Oct 30 00:50:58.088883 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.088760    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fe7cea2165b28cfcd855a7f1f568cf4ca559cb8f63641360bdad7c3f9e5d107"
Oct 30 00:50:58.149183 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.149152    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6"]
Oct 30 00:50:58.149811 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.149796    6099 cpu_manager.go:414] "containerMap: removing container" podUID="d0c42a5f-a350-4683-be19-4884131b1a42" containerName="revoke-keys"
Oct 30 00:50:58.149868 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.149812    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c42a5f-a350-4683-be19-4884131b1a42" containerName="revoke-keys"
Oct 30 00:50:58.149983 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.149972    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c42a5f-a350-4683-be19-4884131b1a42" containerName="revoke-keys"
Oct 30 00:50:58.153500 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.153479    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6"
Oct 30 00:50:58.169234 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.169214    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:50:58.171979 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.171961    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:50:58.175883 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.175862    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:50:58.180208 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.180188    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6"]
Oct 30 00:50:58.270059 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.270036    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:50:58.273764 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.273743    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c5459-8203-4dbe-8cca-64cd23d7aefe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5720f7cd-vfqp6\" (UID: \"d71c5459-8203-4dbe-8cca-64cd23d7aefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6"
Oct 30 00:50:58.273939 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.273919    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c5459-8203-4dbe-8cca-64cd23d7aefe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5720f7cd-vfqp6\" (UID: \"d71c5459-8203-4dbe-8cca-64cd23d7aefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6"
Oct 30 00:50:58.274276 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.274252    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4gpn\" (UniqueName: \"kubernetes.io/projected/d71c5459-8203-4dbe-8cca-64cd23d7aefe-kube-api-access-c4gpn\") pod \"maas-api-revoke-sub-5720f7cd-vfqp6\" (UID: \"d71c5459-8203-4dbe-8cca-64cd23d7aefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6"
Oct 30 00:50:58.376069 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.375995    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4gpn\" (UniqueName: \"kubernetes.io/projected/d71c5459-8203-4dbe-8cca-64cd23d7aefe-kube-api-access-c4gpn\") pod \"maas-api-revoke-sub-5720f7cd-vfqp6\" (UID: \"d71c5459-8203-4dbe-8cca-64cd23d7aefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6"
Oct 30 00:50:58.376498 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.376072    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c5459-8203-4dbe-8cca-64cd23d7aefe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5720f7cd-vfqp6\" (UID: \"d71c5459-8203-4dbe-8cca-64cd23d7aefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6"
Oct 30 00:50:58.376498 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.376120    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c5459-8203-4dbe-8cca-64cd23d7aefe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5720f7cd-vfqp6\" (UID: \"d71c5459-8203-4dbe-8cca-64cd23d7aefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6"
Oct 30 00:50:58.376762 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.376736    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c5459-8203-4dbe-8cca-64cd23d7aefe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5720f7cd-vfqp6\" (UID: \"d71c5459-8203-4dbe-8cca-64cd23d7aefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6"
Oct 30 00:50:58.378418 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.378402    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c5459-8203-4dbe-8cca-64cd23d7aefe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5720f7cd-vfqp6\" (UID: \"d71c5459-8203-4dbe-8cca-64cd23d7aefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6"
Oct 30 00:50:58.401310 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.401282    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4gpn\" (UniqueName: \"kubernetes.io/projected/d71c5459-8203-4dbe-8cca-64cd23d7aefe-kube-api-access-c4gpn\") pod \"maas-api-revoke-sub-5720f7cd-vfqp6\" (UID: \"d71c5459-8203-4dbe-8cca-64cd23d7aefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6"
Oct 30 00:50:58.468466 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.468441    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6"
Oct 30 00:50:58.769898 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:50:58.769857    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71c5459_8203_4dbe_8cca_64cd23d7aefe.slice/crio-08e882e1ba48557c43799e8ab2a70ce00278e4c5c6989a38014209d8da5d7572 WatchSource:0}: Error finding container 08e882e1ba48557c43799e8ab2a70ce00278e4c5c6989a38014209d8da5d7572: Status 404 returned error can't find the container with id 08e882e1ba48557c43799e8ab2a70ce00278e4c5c6989a38014209d8da5d7572
Oct 30 00:50:58.770989 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:58.770960    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6"]
Oct 30 00:50:59.097646 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:59.097615    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6" event={"ID":"d71c5459-8203-4dbe-8cca-64cd23d7aefe","Type":"ContainerStarted","Data":"fea4b2c21da224ed79b3328d8f021273a4be073e20c995eec6febfa6e971e913"}
Oct 30 00:50:59.097801 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:59.097671    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6" event={"ID":"d71c5459-8203-4dbe-8cca-64cd23d7aefe","Type":"ContainerStarted","Data":"08e882e1ba48557c43799e8ab2a70ce00278e4c5c6989a38014209d8da5d7572"}
Oct 30 00:50:59.141074 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:50:59.141029    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6" podStartSLOduration=1.141015564 podStartE2EDuration="1.141015564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:59.136471378 +0000 UTC m=+952.993379132" watchObservedRunningTime="2026-10-30 00:50:59.141015564 +0000 UTC m=+952.997923319"
Oct 30 00:51:00.108227 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:00.107294    6099 generic.go:356] "Generic (PLEG): container finished" podID="d71c5459-8203-4dbe-8cca-64cd23d7aefe" containerID="fea4b2c21da224ed79b3328d8f021273a4be073e20c995eec6febfa6e971e913" exitCode=0
Oct 30 00:51:00.108227 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:00.107508    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6" event={"ID":"d71c5459-8203-4dbe-8cca-64cd23d7aefe","Type":"ContainerDied","Data":"fea4b2c21da224ed79b3328d8f021273a4be073e20c995eec6febfa6e971e913"}
Oct 30 00:51:00.667580 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:00.667534    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv"]
Oct 30 00:51:00.672107 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:00.672084    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv"
Oct 30 00:51:00.691616 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:00.691590    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv"]
Oct 30 00:51:00.809997 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:00.809960    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc01056-d380-4980-b8db-2d2ed16d46fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e68e111f-gzkgv\" (UID: \"fdc01056-d380-4980-b8db-2d2ed16d46fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv"
Oct 30 00:51:00.810147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:00.810024    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc01056-d380-4980-b8db-2d2ed16d46fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e68e111f-gzkgv\" (UID: \"fdc01056-d380-4980-b8db-2d2ed16d46fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv"
Oct 30 00:51:00.810147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:00.810114    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdxrl\" (UniqueName: \"kubernetes.io/projected/fdc01056-d380-4980-b8db-2d2ed16d46fd-kube-api-access-rdxrl\") pod \"maas-api-revoke-sub-e68e111f-gzkgv\" (UID: \"fdc01056-d380-4980-b8db-2d2ed16d46fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv"
Oct 30 00:51:00.911147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:00.911113    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc01056-d380-4980-b8db-2d2ed16d46fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e68e111f-gzkgv\" (UID: \"fdc01056-d380-4980-b8db-2d2ed16d46fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv"
Oct 30 00:51:00.911336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:00.911170    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc01056-d380-4980-b8db-2d2ed16d46fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e68e111f-gzkgv\" (UID: \"fdc01056-d380-4980-b8db-2d2ed16d46fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv"
Oct 30 00:51:00.911336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:00.911210    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdxrl\" (UniqueName: \"kubernetes.io/projected/fdc01056-d380-4980-b8db-2d2ed16d46fd-kube-api-access-rdxrl\") pod \"maas-api-revoke-sub-e68e111f-gzkgv\" (UID: \"fdc01056-d380-4980-b8db-2d2ed16d46fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv"
Oct 30 00:51:00.911854 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:00.911828    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc01056-d380-4980-b8db-2d2ed16d46fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e68e111f-gzkgv\" (UID: \"fdc01056-d380-4980-b8db-2d2ed16d46fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv"
Oct 30 00:51:00.913555 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:00.913534    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc01056-d380-4980-b8db-2d2ed16d46fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e68e111f-gzkgv\" (UID: \"fdc01056-d380-4980-b8db-2d2ed16d46fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv"
Oct 30 00:51:00.934036 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:00.933979    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdxrl\" (UniqueName: \"kubernetes.io/projected/fdc01056-d380-4980-b8db-2d2ed16d46fd-kube-api-access-rdxrl\") pod \"maas-api-revoke-sub-e68e111f-gzkgv\" (UID: \"fdc01056-d380-4980-b8db-2d2ed16d46fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv"
Oct 30 00:51:00.986584 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:00.986554    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv"
Oct 30 00:51:01.281174 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:01.281148    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6"
Oct 30 00:51:01.287652 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:51:01.287613    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc01056_d380_4980_b8db_2d2ed16d46fd.slice/crio-1cae0b4324b7815790953f11e88800174a06b3a6438229ca3c3b8785db140144 WatchSource:0}: Error finding container 1cae0b4324b7815790953f11e88800174a06b3a6438229ca3c3b8785db140144: Status 404 returned error can't find the container with id 1cae0b4324b7815790953f11e88800174a06b3a6438229ca3c3b8785db140144
Oct 30 00:51:01.288855 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:01.288833    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv"]
Oct 30 00:51:01.317548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:01.317519    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d71c5459-8203-4dbe-8cca-64cd23d7aefe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c5459-8203-4dbe-8cca-64cd23d7aefe-maas-api-xks-ca\") pod \"d71c5459-8203-4dbe-8cca-64cd23d7aefe\" (UID: \"d71c5459-8203-4dbe-8cca-64cd23d7aefe\") "
Oct 30 00:51:01.317658 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:01.317584    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71c5459-8203-4dbe-8cca-64cd23d7aefe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c5459-8203-4dbe-8cca-64cd23d7aefe-maas-api-openshift-ca\") pod \"d71c5459-8203-4dbe-8cca-64cd23d7aefe\" (UID: \"d71c5459-8203-4dbe-8cca-64cd23d7aefe\") "
Oct 30 00:51:01.317746 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:01.317721    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71c5459-8203-4dbe-8cca-64cd23d7aefe-kube-api-access-c4gpn\" (UniqueName: \"kubernetes.io/projected/d71c5459-8203-4dbe-8cca-64cd23d7aefe-kube-api-access-c4gpn\") pod \"d71c5459-8203-4dbe-8cca-64cd23d7aefe\" (UID: \"d71c5459-8203-4dbe-8cca-64cd23d7aefe\") "
Oct 30 00:51:01.317979 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:01.317957    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71c5459-8203-4dbe-8cca-64cd23d7aefe-maas-api-openshift-ca" pod "d71c5459-8203-4dbe-8cca-64cd23d7aefe" (UID: "d71c5459-8203-4dbe-8cca-64cd23d7aefe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:51:01.318316 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:01.318298    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c5459-8203-4dbe-8cca-64cd23d7aefe-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:01.319570 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:01.319548    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71c5459-8203-4dbe-8cca-64cd23d7aefe-kube-api-access-c4gpn" pod "d71c5459-8203-4dbe-8cca-64cd23d7aefe" (UID: "d71c5459-8203-4dbe-8cca-64cd23d7aefe"). InnerVolumeSpecName "kube-api-access-c4gpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:01.319718 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:01.319688    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d71c5459-8203-4dbe-8cca-64cd23d7aefe-maas-api-xks-ca" pod "d71c5459-8203-4dbe-8cca-64cd23d7aefe" (UID: "d71c5459-8203-4dbe-8cca-64cd23d7aefe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:51:01.420261 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:01.420229    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c5459-8203-4dbe-8cca-64cd23d7aefe-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:01.420261 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:01.420258    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4gpn\" (UniqueName: \"kubernetes.io/projected/d71c5459-8203-4dbe-8cca-64cd23d7aefe-kube-api-access-c4gpn\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:02.125672 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:02.125638    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv" event={"ID":"fdc01056-d380-4980-b8db-2d2ed16d46fd","Type":"ContainerStarted","Data":"30e0c992865527ccb833c293d94144337463c0b50cabb83eabfdb4947f8dae01"}
Oct 30 00:51:02.125866 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:02.125689    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv" event={"ID":"fdc01056-d380-4980-b8db-2d2ed16d46fd","Type":"ContainerStarted","Data":"1cae0b4324b7815790953f11e88800174a06b3a6438229ca3c3b8785db140144"}
Oct 30 00:51:02.127191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:02.127169    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6"
Oct 30 00:51:02.127311 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:02.127199    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6" event={"ID":"d71c5459-8203-4dbe-8cca-64cd23d7aefe","Type":"ContainerDied","Data":"08e882e1ba48557c43799e8ab2a70ce00278e4c5c6989a38014209d8da5d7572"}
Oct 30 00:51:02.127311 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:02.127236    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08e882e1ba48557c43799e8ab2a70ce00278e4c5c6989a38014209d8da5d7572"
Oct 30 00:51:02.166075 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:02.166032    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv" podStartSLOduration=2.166019558 podStartE2EDuration="2.166019558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:02.164651156 +0000 UTC m=+956.021558913" watchObservedRunningTime="2026-10-30 00:51:02.166019558 +0000 UTC m=+956.022927352"
Oct 30 00:51:03.136607 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.136568    6099 generic.go:356] "Generic (PLEG): container finished" podID="fdc01056-d380-4980-b8db-2d2ed16d46fd" containerID="30e0c992865527ccb833c293d94144337463c0b50cabb83eabfdb4947f8dae01" exitCode=0
Oct 30 00:51:03.137025 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.136648    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv" event={"ID":"fdc01056-d380-4980-b8db-2d2ed16d46fd","Type":"ContainerDied","Data":"30e0c992865527ccb833c293d94144337463c0b50cabb83eabfdb4947f8dae01"}
Oct 30 00:51:03.507858 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.507782    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"]
Oct 30 00:51:03.508414 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.508401    6099 cpu_manager.go:414] "containerMap: removing container" podUID="d71c5459-8203-4dbe-8cca-64cd23d7aefe" containerName="revoke-keys"
Oct 30 00:51:03.508453 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.508415    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71c5459-8203-4dbe-8cca-64cd23d7aefe" containerName="revoke-keys"
Oct 30 00:51:03.508540 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.508531    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71c5459-8203-4dbe-8cca-64cd23d7aefe" containerName="revoke-keys"
Oct 30 00:51:03.512494 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.512479    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.545965 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.545933    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"]
Oct 30 00:51:03.548522 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.548497    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.548672 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.548528    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.548672 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.548562    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vf9g\" (UniqueName: \"kubernetes.io/projected/2bd5a458-625c-4eec-b0df-bbecc83ebed4-kube-api-access-9vf9g\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.548672 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.548634    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.548883 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.548783    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.548883 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.548835    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.548988 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.548900    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.548988 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.548974    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.549102 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.548998    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.653254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.650690    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vf9g\" (UniqueName: \"kubernetes.io/projected/2bd5a458-625c-4eec-b0df-bbecc83ebed4-kube-api-access-9vf9g\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.653254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.650778    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.653254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.650913    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.653254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.650995    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.653254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.651028    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.653254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.651120    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.653254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.651544    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.653254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.651994    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.653254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.652117    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.653254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.652130    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.653254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.652149    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.653254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.652369    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.653254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.652603    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.653254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.653174    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.654985 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.654956    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.655158 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.655138    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.680591 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.679950    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vf9g\" (UniqueName: \"kubernetes.io/projected/2bd5a458-625c-4eec-b0df-bbecc83ebed4-kube-api-access-9vf9g\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.680591 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.680404    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:03.826524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:03.826492    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:04.151166 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.151124    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:51:04.151567 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.151207    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:51:04.151567 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.151255    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:51:04.151567 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.151136    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"]
Oct 30 00:51:04.291924 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.291896    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv"
Oct 30 00:51:04.374123 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.374084    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdc01056-d380-4980-b8db-2d2ed16d46fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc01056-d380-4980-b8db-2d2ed16d46fd-maas-api-xks-ca\") pod \"fdc01056-d380-4980-b8db-2d2ed16d46fd\" (UID: \"fdc01056-d380-4980-b8db-2d2ed16d46fd\") "
Oct 30 00:51:04.375122 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.374520    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc01056-d380-4980-b8db-2d2ed16d46fd-kube-api-access-rdxrl\" (UniqueName: \"kubernetes.io/projected/fdc01056-d380-4980-b8db-2d2ed16d46fd-kube-api-access-rdxrl\") pod \"fdc01056-d380-4980-b8db-2d2ed16d46fd\" (UID: \"fdc01056-d380-4980-b8db-2d2ed16d46fd\") "
Oct 30 00:51:04.375122 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.374691    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdc01056-d380-4980-b8db-2d2ed16d46fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc01056-d380-4980-b8db-2d2ed16d46fd-maas-api-openshift-ca\") pod \"fdc01056-d380-4980-b8db-2d2ed16d46fd\" (UID: \"fdc01056-d380-4980-b8db-2d2ed16d46fd\") "
Oct 30 00:51:04.375355 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.375203    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc01056-d380-4980-b8db-2d2ed16d46fd-maas-api-openshift-ca" pod "fdc01056-d380-4980-b8db-2d2ed16d46fd" (UID: "fdc01056-d380-4980-b8db-2d2ed16d46fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:51:04.376399 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.375910    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc01056-d380-4980-b8db-2d2ed16d46fd-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:04.377984 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.377958    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc01056-d380-4980-b8db-2d2ed16d46fd-maas-api-xks-ca" pod "fdc01056-d380-4980-b8db-2d2ed16d46fd" (UID: "fdc01056-d380-4980-b8db-2d2ed16d46fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:51:04.378098 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.378010    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc01056-d380-4980-b8db-2d2ed16d46fd-kube-api-access-rdxrl" pod "fdc01056-d380-4980-b8db-2d2ed16d46fd" (UID: "fdc01056-d380-4980-b8db-2d2ed16d46fd"). InnerVolumeSpecName "kube-api-access-rdxrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:04.412766 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.409701    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5"]
Oct 30 00:51:04.412766 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.410670    6099 cpu_manager.go:414] "containerMap: removing container" podUID="fdc01056-d380-4980-b8db-2d2ed16d46fd" containerName="revoke-keys"
Oct 30 00:51:04.412766 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.410690    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc01056-d380-4980-b8db-2d2ed16d46fd" containerName="revoke-keys"
Oct 30 00:51:04.412766 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.410905    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc01056-d380-4980-b8db-2d2ed16d46fd" containerName="revoke-keys"
Oct 30 00:51:04.416342 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.416319    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5"
Oct 30 00:51:04.416907 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.416882    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5"]
Oct 30 00:51:04.477180 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.477148    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s78r8\" (UniqueName: \"kubernetes.io/projected/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-kube-api-access-s78r8\") pod \"maas-api-revoke-sub-fd5d58b3-79ch5\" (UID: \"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5"
Oct 30 00:51:04.477315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.477202    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd5d58b3-79ch5\" (UID: \"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5"
Oct 30 00:51:04.477315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.477295    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd5d58b3-79ch5\" (UID: \"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5"
Oct 30 00:51:04.477565 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.477543    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdxrl\" (UniqueName: \"kubernetes.io/projected/fdc01056-d380-4980-b8db-2d2ed16d46fd-kube-api-access-rdxrl\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:04.477599 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.477570    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc01056-d380-4980-b8db-2d2ed16d46fd-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:04.579218 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.579185    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s78r8\" (UniqueName: \"kubernetes.io/projected/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-kube-api-access-s78r8\") pod \"maas-api-revoke-sub-fd5d58b3-79ch5\" (UID: \"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5"
Oct 30 00:51:04.579393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.579237    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd5d58b3-79ch5\" (UID: \"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5"
Oct 30 00:51:04.579393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.579283    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd5d58b3-79ch5\" (UID: \"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5"
Oct 30 00:51:04.580062 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.580037    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd5d58b3-79ch5\" (UID: \"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5"
Oct 30 00:51:04.581694 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.581676    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd5d58b3-79ch5\" (UID: \"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5"
Oct 30 00:51:04.601961 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.601930    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s78r8\" (UniqueName: \"kubernetes.io/projected/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-kube-api-access-s78r8\") pod \"maas-api-revoke-sub-fd5d58b3-79ch5\" (UID: \"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5"
Oct 30 00:51:04.742959 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:04.742928    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5"
Oct 30 00:51:05.044749 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:51:05.044719    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02b4aa7_ea91_4b32_a463_0c1f9fca8e81.slice/crio-59deecb3666f8b72368158ce4a2b35385f8d7563d74c1af7ff54a7d7376dcb9b WatchSource:0}: Error finding container 59deecb3666f8b72368158ce4a2b35385f8d7563d74c1af7ff54a7d7376dcb9b: Status 404 returned error can't find the container with id 59deecb3666f8b72368158ce4a2b35385f8d7563d74c1af7ff54a7d7376dcb9b
Oct 30 00:51:05.045648 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:05.045624    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5"]
Oct 30 00:51:05.155443 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:05.155413    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l" event={"ID":"2bd5a458-625c-4eec-b0df-bbecc83ebed4","Type":"ContainerStarted","Data":"564bb0e684e51330212851e56a666ce6eeee2fb176b3384586b9bf7015a41ff8"}
Oct 30 00:51:05.155877 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:05.155474    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l" event={"ID":"2bd5a458-625c-4eec-b0df-bbecc83ebed4","Type":"ContainerStarted","Data":"29adb050b819d800768ced889270438e78e334d295355de5f30fa77f1982d0fb"}
Oct 30 00:51:05.157005 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:05.156981    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv" event={"ID":"fdc01056-d380-4980-b8db-2d2ed16d46fd","Type":"ContainerDied","Data":"1cae0b4324b7815790953f11e88800174a06b3a6438229ca3c3b8785db140144"}
Oct 30 00:51:05.157005 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:05.156996    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv"
Oct 30 00:51:05.157171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:05.157025    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cae0b4324b7815790953f11e88800174a06b3a6438229ca3c3b8785db140144"
Oct 30 00:51:05.158374 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:05.158351    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5" event={"ID":"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81","Type":"ContainerStarted","Data":"59deecb3666f8b72368158ce4a2b35385f8d7563d74c1af7ff54a7d7376dcb9b"}
Oct 30 00:51:05.202548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:05.202499    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l" podStartSLOduration=2.2024850320000002 podStartE2EDuration="2.202485032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:05.199186858 +0000 UTC m=+959.056094630" watchObservedRunningTime="2026-10-30 00:51:05.202485032 +0000 UTC m=+959.059392787"
Oct 30 00:51:05.827670 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:05.827641    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:05.828977 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:05.828947    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 00:51:05.829032 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:05.828996    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l" podUID="2bd5a458-625c-4eec-b0df-bbecc83ebed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 00:51:06.196364 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:06.196327    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5" event={"ID":"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81","Type":"ContainerStarted","Data":"446bfddbd15b1490cf4dc0c42f06616d8226420b831b3cd1c46d703e172d8a9d"}
Oct 30 00:51:06.827851 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:06.827819    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 00:51:06.828064 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:06.827883    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l" podUID="2bd5a458-625c-4eec-b0df-bbecc83ebed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 00:51:07.205837 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:07.205748    6099 generic.go:356] "Generic (PLEG): container finished" podID="a02b4aa7-ea91-4b32-a463-0c1f9fca8e81" containerID="446bfddbd15b1490cf4dc0c42f06616d8226420b831b3cd1c46d703e172d8a9d" exitCode=0
Oct 30 00:51:07.206199 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:07.205875    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5" event={"ID":"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81","Type":"ContainerDied","Data":"446bfddbd15b1490cf4dc0c42f06616d8226420b831b3cd1c46d703e172d8a9d"}
Oct 30 00:51:07.827396 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:07.827359    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 00:51:07.827572 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:07.827409    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l" podUID="2bd5a458-625c-4eec-b0df-bbecc83ebed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 00:51:08.450554 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:08.450529    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5"
Oct 30 00:51:08.538471 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:08.538388    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-kube-api-access-s78r8\" (UniqueName: \"kubernetes.io/projected/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-kube-api-access-s78r8\") pod \"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81\" (UID: \"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81\") "
Oct 30 00:51:08.538471 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:08.538456    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-maas-api-xks-ca\") pod \"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81\" (UID: \"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81\") "
Oct 30 00:51:08.538842 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:08.538817    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-maas-api-openshift-ca\") pod \"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81\" (UID: \"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81\") "
Oct 30 00:51:08.539222 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:08.539190    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-maas-api-openshift-ca" pod "a02b4aa7-ea91-4b32-a463-0c1f9fca8e81" (UID: "a02b4aa7-ea91-4b32-a463-0c1f9fca8e81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:51:08.539505 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:08.539483    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:08.540733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:08.540684    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-kube-api-access-s78r8" pod "a02b4aa7-ea91-4b32-a463-0c1f9fca8e81" (UID: "a02b4aa7-ea91-4b32-a463-0c1f9fca8e81"). InnerVolumeSpecName "kube-api-access-s78r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:08.541425 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:08.541399    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-maas-api-xks-ca" pod "a02b4aa7-ea91-4b32-a463-0c1f9fca8e81" (UID: "a02b4aa7-ea91-4b32-a463-0c1f9fca8e81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:51:08.641124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:08.641082    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s78r8\" (UniqueName: \"kubernetes.io/projected/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-kube-api-access-s78r8\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:08.641124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:08.641121    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:08.826917 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:08.826885    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 00:51:08.827089 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:08.826935    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l" podUID="2bd5a458-625c-4eec-b0df-bbecc83ebed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 00:51:09.223208 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:09.223131    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5"
Oct 30 00:51:09.223208 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:09.223144    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5" event={"ID":"a02b4aa7-ea91-4b32-a463-0c1f9fca8e81","Type":"ContainerDied","Data":"59deecb3666f8b72368158ce4a2b35385f8d7563d74c1af7ff54a7d7376dcb9b"}
Oct 30 00:51:09.223391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:09.223246    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59deecb3666f8b72368158ce4a2b35385f8d7563d74c1af7ff54a7d7376dcb9b"
Oct 30 00:51:09.827765 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:09.827726    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 00:51:09.828132 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:09.827802    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l" podUID="2bd5a458-625c-4eec-b0df-bbecc83ebed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 00:51:10.828102 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:10.828068    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 00:51:10.828670 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:10.828639    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l" podUID="2bd5a458-625c-4eec-b0df-bbecc83ebed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 00:51:11.191957 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:11.191831    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:11.292267 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:11.291947    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Oct 30 00:51:11.305848 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:11.305802    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 00:51:11.366843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:11.366794    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:11.391825 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:11.391781    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:11.393157 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:11.393131    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:11.393575 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:11.393549    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:51:11.394873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:11.394847    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:11.418871 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:11.412908    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:51:11.418871 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:11.413592    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 00:51:11.418871 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:11.415795    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:11.830075 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:11.830038    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 00:51:11.830640 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:11.830607    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l" podUID="2bd5a458-625c-4eec-b0df-bbecc83ebed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 00:51:12.014837 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.014790    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:12.194937 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.194821    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:12.196014 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.195988    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:51:12.319731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.319010    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:12.461786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.443799    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:12.461786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.445382    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:51:12.577839 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.539025    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 00:51:12.577839 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.540824    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:12.577839 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.540860    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:51:12.577839 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.576297    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 00:51:12.577839 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.540827    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:51:12.600587 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.600556    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:51:12.608060 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.608038    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:51:12.608253 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.608241    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:51:12.608401 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.608388    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:51:12.608544 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.608533    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:51:12.608670 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.608659    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:51:12.608825 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.608812    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:51:12.608941 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.608930    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:51:12.632752 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.541496    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:51:12.632752 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.632216    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:51:12.837869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.835031    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 00:51:12.837869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.835092    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l" podUID="2bd5a458-625c-4eec-b0df-bbecc83ebed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 00:51:12.997731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:12.983950    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:13.004945 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:13.004911    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:51:13.020728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:13.017146    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp" podUID="e5edebd8-443c-42d5-afc8-2c5495b4af01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:13.046731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:13.012147    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:13.211789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:13.195638    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:13.440420 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:13.440360    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 00:51:13.461739 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:13.459656    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:51:13.670731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:13.459889    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 00:51:13.924789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:13.912418    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:51:13.924789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:13.912480    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp" podUID="74025dac-eaf2-43b0-82d0-5470bd908c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:13.924789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:13.831914    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:51:13.924789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:13.809775    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 00:51:13.924789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:13.831981    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 00:51:13.925867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:13.925838    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:51:13.926014 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:13.834876    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 00:51:13.926148 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:13.926117    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd" podUID="96e60f3b-c78f-4de5-aefe-d4e8ee216a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:13.926268 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:13.832004    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:51:13.954221 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:13.680793    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:14.367844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:14.367773    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:14.528317 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:14.437874    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:51:14.555868 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:14.555829    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp" podUID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:14.556115 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:14.462951    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:14.556595 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:14.462835    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:14.742136 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:14.733802    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:51:14.742136 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:14.733867    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7" podUID="ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:14.873741 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:14.851823    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:51:14.873741 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:14.851884    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l" podUID="2bd5a458-625c-4eec-b0df-bbecc83ebed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:14.947760 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:14.920843    6099 patch_prober.go:29] interesting pod/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:51:14.947760 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:14.920907    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml" podUID="c59323d4-d59d-421d-8c1e-00570828759a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:14.947760 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:14.922407    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 00:51:14.988742 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:14.954832    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:14.988742 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:14.956492    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:14.988742 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:14.956821    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:51:14.988742 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:14.957084    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 00:51:14.988742 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:14.957359    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 00:51:14.988742 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:14.957649    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:15.021029 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.020898    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:51:15.198639 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.198504    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl" podUID="90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:15.382082 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.382045    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:15.392129 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.392101    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:15.429753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.415194    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:15.454801 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.453810    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:51:15.472326 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.471789    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:51:15.472326 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.471849    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh" podUID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:15.600690 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.598459    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:51:15.650347 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.648776    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:51:15.715435 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.715390    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:51:15.748569 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.748539    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:51:15.769306 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.769277    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:51:15.772567 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.772525    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:15.774369 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.774349    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:51:15.795980 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.795953    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:51:15.850309 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.850280    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:51:15.865085 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.860472    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:51:15.876464 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:15.876438    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:51:16.629490 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:51:16.629444    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:51:16.660022 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.659990    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv"]
Oct 30 00:51:16.660982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.660962    6099 cpu_manager.go:414] "containerMap: removing container" podUID="a02b4aa7-ea91-4b32-a463-0c1f9fca8e81" containerName="revoke-keys"
Oct 30 00:51:16.661041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.660985    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02b4aa7-ea91-4b32-a463-0c1f9fca8e81" containerName="revoke-keys"
Oct 30 00:51:16.661204 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.661192    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02b4aa7-ea91-4b32-a463-0c1f9fca8e81" containerName="revoke-keys"
Oct 30 00:51:16.666781 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.666758    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv"
Oct 30 00:51:16.690745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.687181    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:51:16.690745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.687645    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv"]
Oct 30 00:51:16.690745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.687819    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:51:16.690745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.687919    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:51:16.690745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.689942    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:51:16.760826 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.760794    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2972b5-c368-4a99-8ca1-7c0f0b271870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13e7f344-vj8wv\" (UID: \"df2972b5-c368-4a99-8ca1-7c0f0b271870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv"
Oct 30 00:51:16.761018 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.760994    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvfb5\" (UniqueName: \"kubernetes.io/projected/df2972b5-c368-4a99-8ca1-7c0f0b271870-kube-api-access-fvfb5\") pod \"maas-api-revoke-sub-13e7f344-vj8wv\" (UID: \"df2972b5-c368-4a99-8ca1-7c0f0b271870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv"
Oct 30 00:51:16.761080 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.761053    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2972b5-c368-4a99-8ca1-7c0f0b271870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13e7f344-vj8wv\" (UID: \"df2972b5-c368-4a99-8ca1-7c0f0b271870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv"
Oct 30 00:51:16.762549 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.762529    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk"]
Oct 30 00:51:16.767056 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.767037    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk"
Oct 30 00:51:16.786966 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.786940    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk"]
Oct 30 00:51:16.862041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.862012    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2972b5-c368-4a99-8ca1-7c0f0b271870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13e7f344-vj8wv\" (UID: \"df2972b5-c368-4a99-8ca1-7c0f0b271870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv"
Oct 30 00:51:16.862233 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.862132    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1b05b1b-bmvsk\" (UID: \"e78b91bc-deda-4bc0-ac7d-c97e75bf6dce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk"
Oct 30 00:51:16.862233 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.862199    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxfrs\" (UniqueName: \"kubernetes.io/projected/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-kube-api-access-lxfrs\") pod \"maas-api-revoke-sub-c1b05b1b-bmvsk\" (UID: \"e78b91bc-deda-4bc0-ac7d-c97e75bf6dce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk"
Oct 30 00:51:16.862348 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.862230    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1b05b1b-bmvsk\" (UID: \"e78b91bc-deda-4bc0-ac7d-c97e75bf6dce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk"
Oct 30 00:51:16.862402 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.862338    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvfb5\" (UniqueName: \"kubernetes.io/projected/df2972b5-c368-4a99-8ca1-7c0f0b271870-kube-api-access-fvfb5\") pod \"maas-api-revoke-sub-13e7f344-vj8wv\" (UID: \"df2972b5-c368-4a99-8ca1-7c0f0b271870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv"
Oct 30 00:51:16.862402 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.862392    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2972b5-c368-4a99-8ca1-7c0f0b271870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13e7f344-vj8wv\" (UID: \"df2972b5-c368-4a99-8ca1-7c0f0b271870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv"
Oct 30 00:51:16.863129 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.863093    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2972b5-c368-4a99-8ca1-7c0f0b271870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13e7f344-vj8wv\" (UID: \"df2972b5-c368-4a99-8ca1-7c0f0b271870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv"
Oct 30 00:51:16.864799 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.864774    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2972b5-c368-4a99-8ca1-7c0f0b271870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13e7f344-vj8wv\" (UID: \"df2972b5-c368-4a99-8ca1-7c0f0b271870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv"
Oct 30 00:51:16.885482 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.885426    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvfb5\" (UniqueName: \"kubernetes.io/projected/df2972b5-c368-4a99-8ca1-7c0f0b271870-kube-api-access-fvfb5\") pod \"maas-api-revoke-sub-13e7f344-vj8wv\" (UID: \"df2972b5-c368-4a99-8ca1-7c0f0b271870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv"
Oct 30 00:51:16.963520 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.963493    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1b05b1b-bmvsk\" (UID: \"e78b91bc-deda-4bc0-ac7d-c97e75bf6dce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk"
Oct 30 00:51:16.963671 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.963555    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxfrs\" (UniqueName: \"kubernetes.io/projected/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-kube-api-access-lxfrs\") pod \"maas-api-revoke-sub-c1b05b1b-bmvsk\" (UID: \"e78b91bc-deda-4bc0-ac7d-c97e75bf6dce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk"
Oct 30 00:51:16.963671 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.963577    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1b05b1b-bmvsk\" (UID: \"e78b91bc-deda-4bc0-ac7d-c97e75bf6dce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk"
Oct 30 00:51:16.964189 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.964160    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1b05b1b-bmvsk\" (UID: \"e78b91bc-deda-4bc0-ac7d-c97e75bf6dce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk"
Oct 30 00:51:16.965946 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.965925    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1b05b1b-bmvsk\" (UID: \"e78b91bc-deda-4bc0-ac7d-c97e75bf6dce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk"
Oct 30 00:51:16.986458 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.986432    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxfrs\" (UniqueName: \"kubernetes.io/projected/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-kube-api-access-lxfrs\") pod \"maas-api-revoke-sub-c1b05b1b-bmvsk\" (UID: \"e78b91bc-deda-4bc0-ac7d-c97e75bf6dce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk"
Oct 30 00:51:16.987221 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:16.987204    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv"
Oct 30 00:51:17.083201 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:17.083160    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk"
Oct 30 00:51:17.301053 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:51:17.301023    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2972b5_c368_4a99_8ca1_7c0f0b271870.slice/crio-39c436c6a343249bfb5cfe5f2424101adf6b5b14dd3dd706bd3dc171b47be324 WatchSource:0}: Error finding container 39c436c6a343249bfb5cfe5f2424101adf6b5b14dd3dd706bd3dc171b47be324: Status 404 returned error can't find the container with id 39c436c6a343249bfb5cfe5f2424101adf6b5b14dd3dd706bd3dc171b47be324
Oct 30 00:51:17.302073 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:17.302045    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv"]
Oct 30 00:51:17.399107 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:51:17.399042    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78b91bc_deda_4bc0_ac7d_c97e75bf6dce.slice/crio-d615f80f855d243a74b27691614b65c58861168aff405b5177e25266a8b4e267 WatchSource:0}: Error finding container d615f80f855d243a74b27691614b65c58861168aff405b5177e25266a8b4e267: Status 404 returned error can't find the container with id d615f80f855d243a74b27691614b65c58861168aff405b5177e25266a8b4e267
Oct 30 00:51:17.400916 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:17.400873    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk"]
Oct 30 00:51:17.481288 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:17.481256    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv" event={"ID":"df2972b5-c368-4a99-8ca1-7c0f0b271870","Type":"ContainerStarted","Data":"39c436c6a343249bfb5cfe5f2424101adf6b5b14dd3dd706bd3dc171b47be324"}
Oct 30 00:51:17.482757 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:17.482730    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk" event={"ID":"e78b91bc-deda-4bc0-ac7d-c97e75bf6dce","Type":"ContainerStarted","Data":"d615f80f855d243a74b27691614b65c58861168aff405b5177e25266a8b4e267"}
Oct 30 00:51:18.486967 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.486881    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"]
Oct 30 00:51:18.508092 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.507926    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.527857 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.527825    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"]
Oct 30 00:51:18.530427 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.530398    6099 generic.go:356] "Generic (PLEG): container finished" podID="df2972b5-c368-4a99-8ca1-7c0f0b271870" containerID="b0e6f2894c430e195e91be8a328f22bcab64a8079b8931a960e850d6657f8169" exitCode=0
Oct 30 00:51:18.530566 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.530484    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv" event={"ID":"df2972b5-c368-4a99-8ca1-7c0f0b271870","Type":"ContainerDied","Data":"b0e6f2894c430e195e91be8a328f22bcab64a8079b8931a960e850d6657f8169"}
Oct 30 00:51:18.532687 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.532659    6099 generic.go:356] "Generic (PLEG): container finished" podID="e78b91bc-deda-4bc0-ac7d-c97e75bf6dce" containerID="58f1a83f0686182ce1fa97415bdd49d64486fd55a0ce65a4a623407b5491a7b6" exitCode=0
Oct 30 00:51:18.532848 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.532758    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk" event={"ID":"e78b91bc-deda-4bc0-ac7d-c97e75bf6dce","Type":"ContainerDied","Data":"58f1a83f0686182ce1fa97415bdd49d64486fd55a0ce65a4a623407b5491a7b6"}
Oct 30 00:51:18.693887 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.693847    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.694053 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.693931    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvvw\" (UniqueName: \"kubernetes.io/projected/2fc4ad98-ba46-4c8d-911e-dad3e0617670-kube-api-access-5cvvw\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.694149 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.694083    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.694149 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.694122    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.694230 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.694147    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.694285 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.694259    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.694364 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.694342    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.694400 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.694387    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.694458 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.694440    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.795833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.795801    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.796001 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.795840    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.796001 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.795894    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.796001 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.795944    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.796001 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.795977    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.796001 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.795996    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.796329 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.796074    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.796329 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.796126    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvvw\" (UniqueName: \"kubernetes.io/projected/2fc4ad98-ba46-4c8d-911e-dad3e0617670-kube-api-access-5cvvw\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.796329 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.796230    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.796329 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.796267    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.796592 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.796571    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.796748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.796723    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.796974 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.796952    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.797131 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.797110    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.798349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.798324    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.799040 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.799013    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.819867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.819833    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvvw\" (UniqueName: \"kubernetes.io/projected/2fc4ad98-ba46-4c8d-911e-dad3e0617670-kube-api-access-5cvvw\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.819990 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.819954    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:18.844654 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:18.844626    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:19.171943 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:51:19.171908    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc4ad98_ba46_4c8d_911e_dad3e0617670.slice/crio-2177855259df7dbb1103148f370112f1c25f26e129f6572c7e43c94732c09052 WatchSource:0}: Error finding container 2177855259df7dbb1103148f370112f1c25f26e129f6572c7e43c94732c09052: Status 404 returned error can't find the container with id 2177855259df7dbb1103148f370112f1c25f26e129f6572c7e43c94732c09052
Oct 30 00:51:19.173416 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.173388    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"]
Oct 30 00:51:19.174756 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.174700    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:51:19.174835 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.174799    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:51:19.174871 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.174843    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:51:19.317545 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.317510    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"]
Oct 30 00:51:19.323407 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.323367    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.336858 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.336820    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"]
Oct 30 00:51:19.388729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.387787    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"]
Oct 30 00:51:19.392973 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.392942    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.406013 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.405980    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.406133 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.406031    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.406133 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.406087    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.406133 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.406125    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.406277 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.406197    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.406344 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.406324    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.406429 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.406359    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.406429 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.406421    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.406534 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.406498    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.406642 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.406621    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.406802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.406783    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.406858 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.406842    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.406968 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.406948    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.407039 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.406988    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.407093 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.407044    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kz7n\" (UniqueName: \"kubernetes.io/projected/29d2f6eb-bea1-4aec-baaa-4146db3218e3-kube-api-access-9kz7n\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.407198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.407150    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.407315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.407272    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68djq\" (UniqueName: \"kubernetes.io/projected/f5d7156f-8120-4ab7-83ae-f7ab80171e94-kube-api-access-68djq\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.407315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.407305    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.485015 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.484931    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"]
Oct 30 00:51:19.508222 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.508190    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.508844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.508234    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.508844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.508290    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.508844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.508337    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.508844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.508379    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.508844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.508413    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.508844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.508437    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.508844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.508471    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.508844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.508562    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.508844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.508609    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.508844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.508616    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-credential-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.508844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.508693    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.508844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.508767    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.508844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.508801    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.508844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.508835    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.509724 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.508878    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kz7n\" (UniqueName: \"kubernetes.io/projected/29d2f6eb-bea1-4aec-baaa-4146db3218e3-kube-api-access-9kz7n\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.509724 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.508982    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.509724 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.509007    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.509724 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.509027    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68djq\" (UniqueName: \"kubernetes.io/projected/f5d7156f-8120-4ab7-83ae-f7ab80171e94-kube-api-access-68djq\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.509724 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.509076    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.509724 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.509433    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.509724 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.509558    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.510045 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.509808    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-data\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.510119 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.510052    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-workload-socket\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.510308 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.510286    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-workload-certs\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.511389 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.511367    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.511581 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.511557    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istiod-ca-cert\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.511670 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.511650    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.513493 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.513474    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.513605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.513591    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-envoy\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.514404 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.514382    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.516142 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.516124    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-podinfo\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.542379 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.542346    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kz7n\" (UniqueName: \"kubernetes.io/projected/29d2f6eb-bea1-4aec-baaa-4146db3218e3-kube-api-access-9kz7n\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.543626 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.543598    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.543753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.543625    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-token\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.543832 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.543746    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68djq\" (UniqueName: \"kubernetes.io/projected/f5d7156f-8120-4ab7-83ae-f7ab80171e94-kube-api-access-68djq\") pod \"e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") " pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.549111 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.549078    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt" event={"ID":"2fc4ad98-ba46-4c8d-911e-dad3e0617670","Type":"ContainerStarted","Data":"6645e41fe6276ad0221ca2df67a542faea1b83ff51be74551e39f6e04aa42722"}
Oct 30 00:51:19.549217 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.549138    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt" event={"ID":"2fc4ad98-ba46-4c8d-911e-dad3e0617670","Type":"ContainerStarted","Data":"2177855259df7dbb1103148f370112f1c25f26e129f6572c7e43c94732c09052"}
Oct 30 00:51:19.599656 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.599590    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt" podStartSLOduration=1.599573089 podStartE2EDuration="1.599573089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:19.596344436 +0000 UTC m=+973.453252191" watchObservedRunningTime="2026-10-30 00:51:19.599573089 +0000 UTC m=+973.456480843"
Oct 30 00:51:19.648053 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.647741    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:19.709413 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.709166    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:19.770942 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.770268    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk"
Oct 30 00:51:19.802029 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.801984    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv"
Oct 30 00:51:19.816665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.815312    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df2972b5-c368-4a99-8ca1-7c0f0b271870-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2972b5-c368-4a99-8ca1-7c0f0b271870-maas-api-xks-ca\") pod \"df2972b5-c368-4a99-8ca1-7c0f0b271870\" (UID: \"df2972b5-c368-4a99-8ca1-7c0f0b271870\") "
Oct 30 00:51:19.816665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.815400    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-kube-api-access-lxfrs\" (UniqueName: \"kubernetes.io/projected/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-kube-api-access-lxfrs\") pod \"e78b91bc-deda-4bc0-ac7d-c97e75bf6dce\" (UID: \"e78b91bc-deda-4bc0-ac7d-c97e75bf6dce\") "
Oct 30 00:51:19.816665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.815440    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-maas-api-openshift-ca\") pod \"e78b91bc-deda-4bc0-ac7d-c97e75bf6dce\" (UID: \"e78b91bc-deda-4bc0-ac7d-c97e75bf6dce\") "
Oct 30 00:51:19.816665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.815521    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df2972b5-c368-4a99-8ca1-7c0f0b271870-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2972b5-c368-4a99-8ca1-7c0f0b271870-maas-api-openshift-ca\") pod \"df2972b5-c368-4a99-8ca1-7c0f0b271870\" (UID: \"df2972b5-c368-4a99-8ca1-7c0f0b271870\") "
Oct 30 00:51:19.816665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.815598    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-maas-api-xks-ca\") pod \"e78b91bc-deda-4bc0-ac7d-c97e75bf6dce\" (UID: \"e78b91bc-deda-4bc0-ac7d-c97e75bf6dce\") "
Oct 30 00:51:19.816665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.815622    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2972b5-c368-4a99-8ca1-7c0f0b271870-kube-api-access-fvfb5\" (UniqueName: \"kubernetes.io/projected/df2972b5-c368-4a99-8ca1-7c0f0b271870-kube-api-access-fvfb5\") pod \"df2972b5-c368-4a99-8ca1-7c0f0b271870\" (UID: \"df2972b5-c368-4a99-8ca1-7c0f0b271870\") "
Oct 30 00:51:19.816665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.816741    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-maas-api-openshift-ca" pod "e78b91bc-deda-4bc0-ac7d-c97e75bf6dce" (UID: "e78b91bc-deda-4bc0-ac7d-c97e75bf6dce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:51:19.817943 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.817905    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df2972b5-c368-4a99-8ca1-7c0f0b271870-maas-api-openshift-ca" pod "df2972b5-c368-4a99-8ca1-7c0f0b271870" (UID: "df2972b5-c368-4a99-8ca1-7c0f0b271870"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:51:19.821464 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.821430    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-maas-api-xks-ca" pod "e78b91bc-deda-4bc0-ac7d-c97e75bf6dce" (UID: "e78b91bc-deda-4bc0-ac7d-c97e75bf6dce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:51:19.821749 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.821685    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-kube-api-access-lxfrs" pod "e78b91bc-deda-4bc0-ac7d-c97e75bf6dce" (UID: "e78b91bc-deda-4bc0-ac7d-c97e75bf6dce"). InnerVolumeSpecName "kube-api-access-lxfrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:19.827972 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.827948    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2972b5-c368-4a99-8ca1-7c0f0b271870-kube-api-access-fvfb5" pod "df2972b5-c368-4a99-8ca1-7c0f0b271870" (UID: "df2972b5-c368-4a99-8ca1-7c0f0b271870"). InnerVolumeSpecName "kube-api-access-fvfb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:19.829847 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.829732    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df2972b5-c368-4a99-8ca1-7c0f0b271870-maas-api-xks-ca" pod "df2972b5-c368-4a99-8ca1-7c0f0b271870" (UID: "df2972b5-c368-4a99-8ca1-7c0f0b271870"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:51:19.846061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.845071    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:19.919147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.918260    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2972b5-c368-4a99-8ca1-7c0f0b271870-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:19.919147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.918292    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxfrs\" (UniqueName: \"kubernetes.io/projected/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-kube-api-access-lxfrs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:19.919147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.918307    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:19.919147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.918321    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2972b5-c368-4a99-8ca1-7c0f0b271870-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:19.919147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.918335    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:19.919147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.918348    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvfb5\" (UniqueName: \"kubernetes.io/projected/df2972b5-c368-4a99-8ca1-7c0f0b271870-kube-api-access-fvfb5\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:19.998361 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:19.998268    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"]
Oct 30 00:51:20.009964 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:51:20.009929    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d7156f_8120_4ab7_83ae_f7ab80171e94.slice/crio-ff9995f562d6cc6a3b4eb4074e10548aad8eae122d7c7f39bc2f082c45c8620d WatchSource:0}: Error finding container ff9995f562d6cc6a3b4eb4074e10548aad8eae122d7c7f39bc2f082c45c8620d: Status 404 returned error can't find the container with id ff9995f562d6cc6a3b4eb4074e10548aad8eae122d7c7f39bc2f082c45c8620d
Oct 30 00:51:20.016516 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.015988    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:51:20.017388 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.016727    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:51:20.017388 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.016801    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:51:20.132950 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:51:20.132919    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d2f6eb_bea1_4aec_baaa_4146db3218e3.slice/crio-b3d38fbee0fb93017a783d85735c63a4ede70f86980a61a9f7922ef85e7648e6 WatchSource:0}: Error finding container b3d38fbee0fb93017a783d85735c63a4ede70f86980a61a9f7922ef85e7648e6: Status 404 returned error can't find the container with id b3d38fbee0fb93017a783d85735c63a4ede70f86980a61a9f7922ef85e7648e6
Oct 30 00:51:20.133057 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.133004    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"]
Oct 30 00:51:20.135929 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.135891    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:51:20.136034 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.136000    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:51:20.136093 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.136045    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:51:20.561528 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.561482    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c" event={"ID":"29d2f6eb-bea1-4aec-baaa-4146db3218e3","Type":"ContainerStarted","Data":"9582410bc1d63b2aff5b1ac9e4c3bc1cb9d47715b74e2d0cf56f124d316d4194"}
Oct 30 00:51:20.562040 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.561554    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c" event={"ID":"29d2f6eb-bea1-4aec-baaa-4146db3218e3","Type":"ContainerStarted","Data":"b3d38fbee0fb93017a783d85735c63a4ede70f86980a61a9f7922ef85e7648e6"}
Oct 30 00:51:20.564308 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.564246    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk" event={"ID":"e78b91bc-deda-4bc0-ac7d-c97e75bf6dce","Type":"ContainerDied","Data":"d615f80f855d243a74b27691614b65c58861168aff405b5177e25266a8b4e267"}
Oct 30 00:51:20.564308 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.564268    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk"
Oct 30 00:51:20.564308 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.564298    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d615f80f855d243a74b27691614b65c58861168aff405b5177e25266a8b4e267"
Oct 30 00:51:20.593727 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.584327    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s" event={"ID":"f5d7156f-8120-4ab7-83ae-f7ab80171e94","Type":"ContainerStarted","Data":"bb92c0feca22e00fd02e56c5a70c866a75ca036759acd41c0e303861b42a5270"}
Oct 30 00:51:20.593727 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.584390    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s" event={"ID":"f5d7156f-8120-4ab7-83ae-f7ab80171e94","Type":"ContainerStarted","Data":"ff9995f562d6cc6a3b4eb4074e10548aad8eae122d7c7f39bc2f082c45c8620d"}
Oct 30 00:51:20.603915 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.603873    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv" event={"ID":"df2972b5-c368-4a99-8ca1-7c0f0b271870","Type":"ContainerDied","Data":"39c436c6a343249bfb5cfe5f2424101adf6b5b14dd3dd706bd3dc171b47be324"}
Oct 30 00:51:20.604073 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.603937    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39c436c6a343249bfb5cfe5f2424101adf6b5b14dd3dd706bd3dc171b47be324"
Oct 30 00:51:20.607151 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.607110    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv"
Oct 30 00:51:20.620779 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.616833    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c" podStartSLOduration=1.6168156 podStartE2EDuration="1.6168156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:20.612308904 +0000 UTC m=+974.469216659" watchObservedRunningTime="2026-10-30 00:51:20.6168156 +0000 UTC m=+974.473723357"
Oct 30 00:51:20.648488 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.648457    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:20.686729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.676008    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s" podStartSLOduration=1.675990804 podStartE2EDuration="1.675990804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:20.659954023 +0000 UTC m=+974.516861779" watchObservedRunningTime="2026-10-30 00:51:20.675990804 +0000 UTC m=+974.532898559"
Oct 30 00:51:20.710411 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.710379    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:20.847082 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.846992    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 00:51:20.847082 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:20.847064    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt" podUID="2fc4ad98-ba46-4c8d-911e-dad3e0617670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 00:51:21.545742 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.545690    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6"]
Oct 30 00:51:21.547009 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.546968    6099 cpu_manager.go:414] "containerMap: removing container" podUID="df2972b5-c368-4a99-8ca1-7c0f0b271870" containerName="revoke-keys"
Oct 30 00:51:21.547176 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.547161    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2972b5-c368-4a99-8ca1-7c0f0b271870" containerName="revoke-keys"
Oct 30 00:51:21.547346 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.547335    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e78b91bc-deda-4bc0-ac7d-c97e75bf6dce" containerName="revoke-keys"
Oct 30 00:51:21.547425 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.547415    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78b91bc-deda-4bc0-ac7d-c97e75bf6dce" containerName="revoke-keys"
Oct 30 00:51:21.547675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.547662    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78b91bc-deda-4bc0-ac7d-c97e75bf6dce" containerName="revoke-keys"
Oct 30 00:51:21.547799 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.547786    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2972b5-c368-4a99-8ca1-7c0f0b271870" containerName="revoke-keys"
Oct 30 00:51:21.553860 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.553838    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6"]
Oct 30 00:51:21.554030 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.553955    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6"
Oct 30 00:51:21.592730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.569637    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:51:21.593264 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.588011    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:51:21.593387 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.588063    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:51:21.593465 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.588195    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:51:21.649152 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.649113    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 00:51:21.649353 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.649173    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s" podUID="f5d7156f-8120-4ab7-83ae-f7ab80171e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 00:51:21.744732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.744335    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 00:51:21.744732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.744411    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c" podUID="29d2f6eb-bea1-4aec-baaa-4146db3218e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 00:51:21.746682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.746651    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss8c8\" (UniqueName: \"kubernetes.io/projected/9fb63c91-22fa-45a4-ac81-542c32cb95f7-kube-api-access-ss8c8\") pod \"maas-api-revoke-sub-dd583182-5g6n6\" (UID: \"9fb63c91-22fa-45a4-ac81-542c32cb95f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6"
Oct 30 00:51:21.746862 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.746829    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb63c91-22fa-45a4-ac81-542c32cb95f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd583182-5g6n6\" (UID: \"9fb63c91-22fa-45a4-ac81-542c32cb95f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6"
Oct 30 00:51:21.746947 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.746889    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb63c91-22fa-45a4-ac81-542c32cb95f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd583182-5g6n6\" (UID: \"9fb63c91-22fa-45a4-ac81-542c32cb95f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6"
Oct 30 00:51:21.852366 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.852286    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss8c8\" (UniqueName: \"kubernetes.io/projected/9fb63c91-22fa-45a4-ac81-542c32cb95f7-kube-api-access-ss8c8\") pod \"maas-api-revoke-sub-dd583182-5g6n6\" (UID: \"9fb63c91-22fa-45a4-ac81-542c32cb95f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6"
Oct 30 00:51:21.852650 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.852630    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb63c91-22fa-45a4-ac81-542c32cb95f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd583182-5g6n6\" (UID: \"9fb63c91-22fa-45a4-ac81-542c32cb95f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6"
Oct 30 00:51:21.852863 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.852845    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb63c91-22fa-45a4-ac81-542c32cb95f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd583182-5g6n6\" (UID: \"9fb63c91-22fa-45a4-ac81-542c32cb95f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6"
Oct 30 00:51:21.855169 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.853837    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb63c91-22fa-45a4-ac81-542c32cb95f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd583182-5g6n6\" (UID: \"9fb63c91-22fa-45a4-ac81-542c32cb95f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6"
Oct 30 00:51:21.879729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.856215    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 00:51:21.879729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.856272    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt" podUID="2fc4ad98-ba46-4c8d-911e-dad3e0617670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 00:51:21.924731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.921827    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss8c8\" (UniqueName: \"kubernetes.io/projected/9fb63c91-22fa-45a4-ac81-542c32cb95f7-kube-api-access-ss8c8\") pod \"maas-api-revoke-sub-dd583182-5g6n6\" (UID: \"9fb63c91-22fa-45a4-ac81-542c32cb95f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6"
Oct 30 00:51:21.960876 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:21.960423    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb63c91-22fa-45a4-ac81-542c32cb95f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd583182-5g6n6\" (UID: \"9fb63c91-22fa-45a4-ac81-542c32cb95f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6"
Oct 30 00:51:22.202783 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:22.201989    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6"
Oct 30 00:51:22.653129 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:22.653042    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 00:51:22.653475 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:22.653132    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s" podUID="f5d7156f-8120-4ab7-83ae-f7ab80171e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 00:51:22.746607 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:22.746213    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 00:51:22.750221 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:22.750189    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c" podUID="29d2f6eb-bea1-4aec-baaa-4146db3218e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 00:51:22.753585 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:22.753518    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6"]
Oct 30 00:51:22.761302 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:22.760698    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4"]
Oct 30 00:51:22.902731 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:51:22.880782    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb63c91_22fa_45a4_ac81_542c32cb95f7.slice/crio-3c9daf3060a3bb7bff1787d121e53d8cf43b5b083fc5d2b44d41f982d930effc WatchSource:0}: Error finding container 3c9daf3060a3bb7bff1787d121e53d8cf43b5b083fc5d2b44d41f982d930effc: Status 404 returned error can't find the container with id 3c9daf3060a3bb7bff1787d121e53d8cf43b5b083fc5d2b44d41f982d930effc
Oct 30 00:51:22.902731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:22.881588    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 00:51:22.902731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:22.881638    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt" podUID="2fc4ad98-ba46-4c8d-911e-dad3e0617670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 00:51:22.986068 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:22.986032    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4"
Oct 30 00:51:23.033666 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:23.033581    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4"]
Oct 30 00:51:23.099512 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:23.098997    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b0aa6f6-krmz4\" (UID: \"a8761662-0447-4cbc-bd62-a4d21a0b2ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4"
Oct 30 00:51:23.099512 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:23.099197    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b0aa6f6-krmz4\" (UID: \"a8761662-0447-4cbc-bd62-a4d21a0b2ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4"
Oct 30 00:51:23.099512 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:23.099231    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzmzf\" (UniqueName: \"kubernetes.io/projected/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-kube-api-access-dzmzf\") pod \"maas-api-revoke-sub-6b0aa6f6-krmz4\" (UID: \"a8761662-0447-4cbc-bd62-a4d21a0b2ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4"
Oct 30 00:51:23.212063 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:23.205784    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b0aa6f6-krmz4\" (UID: \"a8761662-0447-4cbc-bd62-a4d21a0b2ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4"
Oct 30 00:51:23.237747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:23.229898    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b0aa6f6-krmz4\" (UID: \"a8761662-0447-4cbc-bd62-a4d21a0b2ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4"
Oct 30 00:51:23.237747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:23.229945    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzmzf\" (UniqueName: \"kubernetes.io/projected/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-kube-api-access-dzmzf\") pod \"maas-api-revoke-sub-6b0aa6f6-krmz4\" (UID: \"a8761662-0447-4cbc-bd62-a4d21a0b2ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4"
Oct 30 00:51:23.310831 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:23.293280    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b0aa6f6-krmz4\" (UID: \"a8761662-0447-4cbc-bd62-a4d21a0b2ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4"
Oct 30 00:51:23.496729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:23.495930    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzmzf\" (UniqueName: \"kubernetes.io/projected/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-kube-api-access-dzmzf\") pod \"maas-api-revoke-sub-6b0aa6f6-krmz4\" (UID: \"a8761662-0447-4cbc-bd62-a4d21a0b2ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4"
Oct 30 00:51:23.564012 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:23.563885    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b0aa6f6-krmz4\" (UID: \"a8761662-0447-4cbc-bd62-a4d21a0b2ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4"
Oct 30 00:51:23.569303 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:23.568932    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4"
Oct 30 00:51:23.648524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:23.648096    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 00:51:23.648524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:23.648154    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s" podUID="f5d7156f-8120-4ab7-83ae-f7ab80171e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 00:51:23.736730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:23.722089    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 00:51:23.736730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:23.722161    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c" podUID="29d2f6eb-bea1-4aec-baaa-4146db3218e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 00:51:23.796116 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:23.782933    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6" event={"ID":"9fb63c91-22fa-45a4-ac81-542c32cb95f7","Type":"ContainerStarted","Data":"3c9daf3060a3bb7bff1787d121e53d8cf43b5b083fc5d2b44d41f982d930effc"}
Oct 30 00:51:23.857663 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:23.856482    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 00:51:23.857663 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:23.856539    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt" podUID="2fc4ad98-ba46-4c8d-911e-dad3e0617670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 00:51:24.353006 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:24.352970    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4"]
Oct 30 00:51:24.649859 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:24.648973    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 00:51:24.649859 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:24.649023    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s" podUID="f5d7156f-8120-4ab7-83ae-f7ab80171e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 00:51:24.732725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:24.723843    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 00:51:24.732725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:24.723907    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c" podUID="29d2f6eb-bea1-4aec-baaa-4146db3218e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 00:51:24.882782 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:24.870553    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 00:51:24.882782 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:24.870611    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt" podUID="2fc4ad98-ba46-4c8d-911e-dad3e0617670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 00:51:25.036772 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:25.025039    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4" event={"ID":"a8761662-0447-4cbc-bd62-a4d21a0b2ab5","Type":"ContainerStarted","Data":"e8b9953317bd0cdf9211045cdff0c95fbb769dff3eb5992abca67e6b20c6000c"}
Oct 30 00:51:25.036772 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:25.028048    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6" event={"ID":"9fb63c91-22fa-45a4-ac81-542c32cb95f7","Type":"ContainerStarted","Data":"fdca38c392537e1e0eaa574da409a58199386ebdb2868b1b72ac34c9725e2d34"}
Oct 30 00:51:25.648619 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:25.648585    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 00:51:25.648831 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:25.648644    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s" podUID="f5d7156f-8120-4ab7-83ae-f7ab80171e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 00:51:25.722750 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:25.722498    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 00:51:25.722750 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:25.722572    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c" podUID="29d2f6eb-bea1-4aec-baaa-4146db3218e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 00:51:25.845231 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:25.845192    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 00:51:25.845428 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:25.845257    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt" podUID="2fc4ad98-ba46-4c8d-911e-dad3e0617670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 00:51:26.038163 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:26.038071    6099 generic.go:356] "Generic (PLEG): container finished" podID="a8761662-0447-4cbc-bd62-a4d21a0b2ab5" containerID="880a519ab11f2d5a7dce880949572d9fc0dd602e7e3d45f905da359267d8d15b" exitCode=0
Oct 30 00:51:26.038163 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:26.038134    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4" event={"ID":"a8761662-0447-4cbc-bd62-a4d21a0b2ab5","Type":"ContainerDied","Data":"880a519ab11f2d5a7dce880949572d9fc0dd602e7e3d45f905da359267d8d15b"}
Oct 30 00:51:26.039815 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:26.039795    6099 generic.go:356] "Generic (PLEG): container finished" podID="9fb63c91-22fa-45a4-ac81-542c32cb95f7" containerID="fdca38c392537e1e0eaa574da409a58199386ebdb2868b1b72ac34c9725e2d34" exitCode=0
Oct 30 00:51:26.040013 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:26.039832    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6" event={"ID":"9fb63c91-22fa-45a4-ac81-542c32cb95f7","Type":"ContainerDied","Data":"fdca38c392537e1e0eaa574da409a58199386ebdb2868b1b72ac34c9725e2d34"}
Oct 30 00:51:26.108031 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:26.107968    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6" podStartSLOduration=5.107947591 podStartE2EDuration="5.107947591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:25.118331346 +0000 UTC m=+978.975239102" watchObservedRunningTime="2026-10-30 00:51:26.107947591 +0000 UTC m=+979.964855348"
Oct 30 00:51:26.648818 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:26.648576    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 00:51:26.648818 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:26.648657    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s" podUID="f5d7156f-8120-4ab7-83ae-f7ab80171e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 00:51:26.710514 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:26.710476    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 00:51:26.710674 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:26.710548    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c" podUID="29d2f6eb-bea1-4aec-baaa-4146db3218e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 00:51:26.997026 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:26.996929    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:26.998340 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:26.998315    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:26.999433 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:26.999410    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:51:27.325257 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.325229    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6"
Oct 30 00:51:27.328759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.328668    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4"
Oct 30 00:51:27.380268 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.380230    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-maas-api-xks-ca\") pod \"a8761662-0447-4cbc-bd62-a4d21a0b2ab5\" (UID: \"a8761662-0447-4cbc-bd62-a4d21a0b2ab5\") "
Oct 30 00:51:27.380426 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.380335    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fb63c91-22fa-45a4-ac81-542c32cb95f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb63c91-22fa-45a4-ac81-542c32cb95f7-maas-api-openshift-ca\") pod \"9fb63c91-22fa-45a4-ac81-542c32cb95f7\" (UID: \"9fb63c91-22fa-45a4-ac81-542c32cb95f7\") "
Oct 30 00:51:27.380426 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.380412    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-kube-api-access-dzmzf\" (UniqueName: \"kubernetes.io/projected/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-kube-api-access-dzmzf\") pod \"a8761662-0447-4cbc-bd62-a4d21a0b2ab5\" (UID: \"a8761662-0447-4cbc-bd62-a4d21a0b2ab5\") "
Oct 30 00:51:27.380611 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.380591    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb63c91-22fa-45a4-ac81-542c32cb95f7-kube-api-access-ss8c8\" (UniqueName: \"kubernetes.io/projected/9fb63c91-22fa-45a4-ac81-542c32cb95f7-kube-api-access-ss8c8\") pod \"9fb63c91-22fa-45a4-ac81-542c32cb95f7\" (UID: \"9fb63c91-22fa-45a4-ac81-542c32cb95f7\") "
Oct 30 00:51:27.380683 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.380645    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-maas-api-openshift-ca\") pod \"a8761662-0447-4cbc-bd62-a4d21a0b2ab5\" (UID: \"a8761662-0447-4cbc-bd62-a4d21a0b2ab5\") "
Oct 30 00:51:27.380778 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.380688    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fb63c91-22fa-45a4-ac81-542c32cb95f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb63c91-22fa-45a4-ac81-542c32cb95f7-maas-api-xks-ca\") pod \"9fb63c91-22fa-45a4-ac81-542c32cb95f7\" (UID: \"9fb63c91-22fa-45a4-ac81-542c32cb95f7\") "
Oct 30 00:51:27.381544 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.381511    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb63c91-22fa-45a4-ac81-542c32cb95f7-maas-api-openshift-ca" pod "9fb63c91-22fa-45a4-ac81-542c32cb95f7" (UID: "9fb63c91-22fa-45a4-ac81-542c32cb95f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:51:27.381544 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.381514    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-maas-api-openshift-ca" pod "a8761662-0447-4cbc-bd62-a4d21a0b2ab5" (UID: "a8761662-0447-4cbc-bd62-a4d21a0b2ab5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:51:27.383463 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.383418    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-maas-api-xks-ca" pod "a8761662-0447-4cbc-bd62-a4d21a0b2ab5" (UID: "a8761662-0447-4cbc-bd62-a4d21a0b2ab5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:51:27.383987 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.383962    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb63c91-22fa-45a4-ac81-542c32cb95f7-kube-api-access-ss8c8" pod "9fb63c91-22fa-45a4-ac81-542c32cb95f7" (UID: "9fb63c91-22fa-45a4-ac81-542c32cb95f7"). InnerVolumeSpecName "kube-api-access-ss8c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:27.384397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.384371    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-kube-api-access-dzmzf" pod "a8761662-0447-4cbc-bd62-a4d21a0b2ab5" (UID: "a8761662-0447-4cbc-bd62-a4d21a0b2ab5"). InnerVolumeSpecName "kube-api-access-dzmzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:27.385165 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.385140    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb63c91-22fa-45a4-ac81-542c32cb95f7-maas-api-xks-ca" pod "9fb63c91-22fa-45a4-ac81-542c32cb95f7" (UID: "9fb63c91-22fa-45a4-ac81-542c32cb95f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:51:27.483033 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.482990    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss8c8\" (UniqueName: \"kubernetes.io/projected/9fb63c91-22fa-45a4-ac81-542c32cb95f7-kube-api-access-ss8c8\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:27.483033 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.483019    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:27.483033 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.483030    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb63c91-22fa-45a4-ac81-542c32cb95f7-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:27.483033 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.483041    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:27.483393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.483049    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb63c91-22fa-45a4-ac81-542c32cb95f7-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:27.483393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.483058    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzmzf\" (UniqueName: \"kubernetes.io/projected/a8761662-0447-4cbc-bd62-a4d21a0b2ab5-kube-api-access-dzmzf\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:27.648203 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.648166    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 00:51:27.648383 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:27.648216    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s" podUID="f5d7156f-8120-4ab7-83ae-f7ab80171e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 00:51:28.064665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:28.064636    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4"
Oct 30 00:51:28.064934 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:28.064652    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4" event={"ID":"a8761662-0447-4cbc-bd62-a4d21a0b2ab5","Type":"ContainerDied","Data":"e8b9953317bd0cdf9211045cdff0c95fbb769dff3eb5992abca67e6b20c6000c"}
Oct 30 00:51:28.065019 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:28.064973    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8b9953317bd0cdf9211045cdff0c95fbb769dff3eb5992abca67e6b20c6000c"
Oct 30 00:51:28.066790 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:28.066760    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6" event={"ID":"9fb63c91-22fa-45a4-ac81-542c32cb95f7","Type":"ContainerDied","Data":"3c9daf3060a3bb7bff1787d121e53d8cf43b5b083fc5d2b44d41f982d930effc"}
Oct 30 00:51:28.066901 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:28.066813    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c9daf3060a3bb7bff1787d121e53d8cf43b5b083fc5d2b44d41f982d930effc"
Oct 30 00:51:28.066901 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:28.066861    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6"
Oct 30 00:51:28.289305 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:28.287833    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:28.289305 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:28.289172    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:28.290240 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:28.290208    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:51:28.656033 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:28.656004    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:28.656491 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:28.656468    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:28.657137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:28.657116    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 00:51:29.702725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.702681    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76"]
Oct 30 00:51:29.703371 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.703355    6099 cpu_manager.go:414] "containerMap: removing container" podUID="9fb63c91-22fa-45a4-ac81-542c32cb95f7" containerName="revoke-keys"
Oct 30 00:51:29.703426 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.703372    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb63c91-22fa-45a4-ac81-542c32cb95f7" containerName="revoke-keys"
Oct 30 00:51:29.703426 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.703382    6099 cpu_manager.go:414] "containerMap: removing container" podUID="a8761662-0447-4cbc-bd62-a4d21a0b2ab5" containerName="revoke-keys"
Oct 30 00:51:29.703426 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.703388    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8761662-0447-4cbc-bd62-a4d21a0b2ab5" containerName="revoke-keys"
Oct 30 00:51:29.703539 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.703530    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb63c91-22fa-45a4-ac81-542c32cb95f7" containerName="revoke-keys"
Oct 30 00:51:29.703577 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.703542    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8761662-0447-4cbc-bd62-a4d21a0b2ab5" containerName="revoke-keys"
Oct 30 00:51:29.708000 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.707981    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76"
Oct 30 00:51:29.723429 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.723405    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:51:29.735259 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.735240    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:51:29.735348 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.735258    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:51:29.735348 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.735307    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:51:29.793325 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.793297    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76"]
Oct 30 00:51:29.831456 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.831423    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97377e3-1b2f-4f3e-be46-b6511041b68c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dbd53e5-g5z76\" (UID: \"a97377e3-1b2f-4f3e-be46-b6511041b68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76"
Oct 30 00:51:29.831643 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.831627    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsjtd\" (UniqueName: \"kubernetes.io/projected/a97377e3-1b2f-4f3e-be46-b6511041b68c-kube-api-access-qsjtd\") pod \"maas-api-revoke-sub-2dbd53e5-g5z76\" (UID: \"a97377e3-1b2f-4f3e-be46-b6511041b68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76"
Oct 30 00:51:29.831737 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.831688    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97377e3-1b2f-4f3e-be46-b6511041b68c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dbd53e5-g5z76\" (UID: \"a97377e3-1b2f-4f3e-be46-b6511041b68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76"
Oct 30 00:51:29.933359 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.933327    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsjtd\" (UniqueName: \"kubernetes.io/projected/a97377e3-1b2f-4f3e-be46-b6511041b68c-kube-api-access-qsjtd\") pod \"maas-api-revoke-sub-2dbd53e5-g5z76\" (UID: \"a97377e3-1b2f-4f3e-be46-b6511041b68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76"
Oct 30 00:51:29.933523 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.933422    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97377e3-1b2f-4f3e-be46-b6511041b68c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dbd53e5-g5z76\" (UID: \"a97377e3-1b2f-4f3e-be46-b6511041b68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76"
Oct 30 00:51:29.933600 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.933520    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97377e3-1b2f-4f3e-be46-b6511041b68c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dbd53e5-g5z76\" (UID: \"a97377e3-1b2f-4f3e-be46-b6511041b68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76"
Oct 30 00:51:29.934645 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.934467    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97377e3-1b2f-4f3e-be46-b6511041b68c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dbd53e5-g5z76\" (UID: \"a97377e3-1b2f-4f3e-be46-b6511041b68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76"
Oct 30 00:51:29.936549 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.936524    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97377e3-1b2f-4f3e-be46-b6511041b68c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dbd53e5-g5z76\" (UID: \"a97377e3-1b2f-4f3e-be46-b6511041b68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76"
Oct 30 00:51:29.956354 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:29.956297    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsjtd\" (UniqueName: \"kubernetes.io/projected/a97377e3-1b2f-4f3e-be46-b6511041b68c-kube-api-access-qsjtd\") pod \"maas-api-revoke-sub-2dbd53e5-g5z76\" (UID: \"a97377e3-1b2f-4f3e-be46-b6511041b68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76"
Oct 30 00:51:30.025733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:30.025693    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76"
Oct 30 00:51:30.332856 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:51:30.332812    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97377e3_1b2f_4f3e_be46_b6511041b68c.slice/crio-e37855354fce18e6dad92a68719fb3f2e4b6a770bd891793d44d756aa5c4feab WatchSource:0}: Error finding container e37855354fce18e6dad92a68719fb3f2e4b6a770bd891793d44d756aa5c4feab: Status 404 returned error can't find the container with id e37855354fce18e6dad92a68719fb3f2e4b6a770bd891793d44d756aa5c4feab
Oct 30 00:51:30.335203 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:30.335156    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76"]
Oct 30 00:51:31.094981 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:31.094935    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76" event={"ID":"a97377e3-1b2f-4f3e-be46-b6511041b68c","Type":"ContainerStarted","Data":"efae10d0ac8aab478c9bcbbc154e600e6a4a1fa3234b1faf041a4ac7c9b986cb"}
Oct 30 00:51:31.095430 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:31.095045    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76" event={"ID":"a97377e3-1b2f-4f3e-be46-b6511041b68c","Type":"ContainerStarted","Data":"e37855354fce18e6dad92a68719fb3f2e4b6a770bd891793d44d756aa5c4feab"}
Oct 30 00:51:31.136214 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:31.136160    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76" podStartSLOduration=2.13614761 podStartE2EDuration="2.13614761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:31.134554394 +0000 UTC m=+984.991462160" watchObservedRunningTime="2026-10-30 00:51:31.13614761 +0000 UTC m=+984.993055365"
Oct 30 00:51:31.945721 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:31.945674    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498"]
Oct 30 00:51:31.950636 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:31.950615    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498"
Oct 30 00:51:31.969302 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:31.969277    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498"]
Oct 30 00:51:32.074575 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:32.074544    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbljp\" (UniqueName: \"kubernetes.io/projected/f50d987e-97c1-4490-afd7-aa57f1b8561f-kube-api-access-gbljp\") pod \"maas-api-revoke-sub-01ad393e-qt498\" (UID: \"f50d987e-97c1-4490-afd7-aa57f1b8561f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498"
Oct 30 00:51:32.074741 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:32.074613    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f50d987e-97c1-4490-afd7-aa57f1b8561f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01ad393e-qt498\" (UID: \"f50d987e-97c1-4490-afd7-aa57f1b8561f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498"
Oct 30 00:51:32.074741 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:32.074695    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f50d987e-97c1-4490-afd7-aa57f1b8561f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01ad393e-qt498\" (UID: \"f50d987e-97c1-4490-afd7-aa57f1b8561f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498"
Oct 30 00:51:32.105445 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:32.105412    6099 generic.go:356] "Generic (PLEG): container finished" podID="a97377e3-1b2f-4f3e-be46-b6511041b68c" containerID="efae10d0ac8aab478c9bcbbc154e600e6a4a1fa3234b1faf041a4ac7c9b986cb" exitCode=0
Oct 30 00:51:32.105830 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:32.105495    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76" event={"ID":"a97377e3-1b2f-4f3e-be46-b6511041b68c","Type":"ContainerDied","Data":"efae10d0ac8aab478c9bcbbc154e600e6a4a1fa3234b1faf041a4ac7c9b986cb"}
Oct 30 00:51:32.176631 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:32.176594    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f50d987e-97c1-4490-afd7-aa57f1b8561f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01ad393e-qt498\" (UID: \"f50d987e-97c1-4490-afd7-aa57f1b8561f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498"
Oct 30 00:51:32.176914 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:32.176884    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f50d987e-97c1-4490-afd7-aa57f1b8561f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01ad393e-qt498\" (UID: \"f50d987e-97c1-4490-afd7-aa57f1b8561f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498"
Oct 30 00:51:32.177299 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:32.177250    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbljp\" (UniqueName: \"kubernetes.io/projected/f50d987e-97c1-4490-afd7-aa57f1b8561f-kube-api-access-gbljp\") pod \"maas-api-revoke-sub-01ad393e-qt498\" (UID: \"f50d987e-97c1-4490-afd7-aa57f1b8561f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498"
Oct 30 00:51:32.177448 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:32.177424    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f50d987e-97c1-4490-afd7-aa57f1b8561f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01ad393e-qt498\" (UID: \"f50d987e-97c1-4490-afd7-aa57f1b8561f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498"
Oct 30 00:51:32.181917 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:32.181764    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f50d987e-97c1-4490-afd7-aa57f1b8561f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01ad393e-qt498\" (UID: \"f50d987e-97c1-4490-afd7-aa57f1b8561f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498"
Oct 30 00:51:32.200855 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:32.200788    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbljp\" (UniqueName: \"kubernetes.io/projected/f50d987e-97c1-4490-afd7-aa57f1b8561f-kube-api-access-gbljp\") pod \"maas-api-revoke-sub-01ad393e-qt498\" (UID: \"f50d987e-97c1-4490-afd7-aa57f1b8561f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498"
Oct 30 00:51:32.265591 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:32.265552    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498"
Oct 30 00:51:32.573806 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:51:32.573773    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf50d987e_97c1_4490_afd7_aa57f1b8561f.slice/crio-782ffcf4e35045f323141bde7bc81e13bf5966a4fad9f4ce0e2283fa1a1df786 WatchSource:0}: Error finding container 782ffcf4e35045f323141bde7bc81e13bf5966a4fad9f4ce0e2283fa1a1df786: Status 404 returned error can't find the container with id 782ffcf4e35045f323141bde7bc81e13bf5966a4fad9f4ce0e2283fa1a1df786
Oct 30 00:51:32.575649 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:32.575627    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498"]
Oct 30 00:51:33.116244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:33.116208    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498" event={"ID":"f50d987e-97c1-4490-afd7-aa57f1b8561f","Type":"ContainerStarted","Data":"e085c2d08ad3d2de13e3baa117de8afd71657d923dc189a2cbabf8da6e9673e3"}
Oct 30 00:51:33.116666 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:33.116273    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498" event={"ID":"f50d987e-97c1-4490-afd7-aa57f1b8561f","Type":"ContainerStarted","Data":"782ffcf4e35045f323141bde7bc81e13bf5966a4fad9f4ce0e2283fa1a1df786"}
Oct 30 00:51:33.157663 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:33.157543    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498" podStartSLOduration=2.157529208 podStartE2EDuration="2.157529208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:33.156871817 +0000 UTC m=+987.013779566" watchObservedRunningTime="2026-10-30 00:51:33.157529208 +0000 UTC m=+987.014436962"
Oct 30 00:51:33.271096 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:33.271073    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76"
Oct 30 00:51:33.395972 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:33.395942    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97377e3-1b2f-4f3e-be46-b6511041b68c-kube-api-access-qsjtd\" (UniqueName: \"kubernetes.io/projected/a97377e3-1b2f-4f3e-be46-b6511041b68c-kube-api-access-qsjtd\") pod \"a97377e3-1b2f-4f3e-be46-b6511041b68c\" (UID: \"a97377e3-1b2f-4f3e-be46-b6511041b68c\") "
Oct 30 00:51:33.396124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:33.396050    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a97377e3-1b2f-4f3e-be46-b6511041b68c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97377e3-1b2f-4f3e-be46-b6511041b68c-maas-api-xks-ca\") pod \"a97377e3-1b2f-4f3e-be46-b6511041b68c\" (UID: \"a97377e3-1b2f-4f3e-be46-b6511041b68c\") "
Oct 30 00:51:33.396202 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:33.396132    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a97377e3-1b2f-4f3e-be46-b6511041b68c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97377e3-1b2f-4f3e-be46-b6511041b68c-maas-api-openshift-ca\") pod \"a97377e3-1b2f-4f3e-be46-b6511041b68c\" (UID: \"a97377e3-1b2f-4f3e-be46-b6511041b68c\") "
Oct 30 00:51:33.396532 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:33.396466    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a97377e3-1b2f-4f3e-be46-b6511041b68c-maas-api-openshift-ca" pod "a97377e3-1b2f-4f3e-be46-b6511041b68c" (UID: "a97377e3-1b2f-4f3e-be46-b6511041b68c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:51:33.397060 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:33.397037    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97377e3-1b2f-4f3e-be46-b6511041b68c-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:33.398249 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:33.398225    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a97377e3-1b2f-4f3e-be46-b6511041b68c-maas-api-xks-ca" pod "a97377e3-1b2f-4f3e-be46-b6511041b68c" (UID: "a97377e3-1b2f-4f3e-be46-b6511041b68c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:51:33.398249 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:33.398234    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97377e3-1b2f-4f3e-be46-b6511041b68c-kube-api-access-qsjtd" pod "a97377e3-1b2f-4f3e-be46-b6511041b68c" (UID: "a97377e3-1b2f-4f3e-be46-b6511041b68c"). InnerVolumeSpecName "kube-api-access-qsjtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:33.499034 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:33.499006    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsjtd\" (UniqueName: \"kubernetes.io/projected/a97377e3-1b2f-4f3e-be46-b6511041b68c-kube-api-access-qsjtd\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:33.499034 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:33.499035    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97377e3-1b2f-4f3e-be46-b6511041b68c-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:34.125652 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.125621    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76"
Oct 30 00:51:34.126160 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.125619    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76" event={"ID":"a97377e3-1b2f-4f3e-be46-b6511041b68c","Type":"ContainerDied","Data":"e37855354fce18e6dad92a68719fb3f2e4b6a770bd891793d44d756aa5c4feab"}
Oct 30 00:51:34.126160 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.125774    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e37855354fce18e6dad92a68719fb3f2e4b6a770bd891793d44d756aa5c4feab"
Oct 30 00:51:34.127280 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.127255    6099 generic.go:356] "Generic (PLEG): container finished" podID="f50d987e-97c1-4490-afd7-aa57f1b8561f" containerID="e085c2d08ad3d2de13e3baa117de8afd71657d923dc189a2cbabf8da6e9673e3" exitCode=0
Oct 30 00:51:34.127418 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.127331    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498" event={"ID":"f50d987e-97c1-4490-afd7-aa57f1b8561f","Type":"ContainerDied","Data":"e085c2d08ad3d2de13e3baa117de8afd71657d923dc189a2cbabf8da6e9673e3"}
Oct 30 00:51:34.407654 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.407567    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"]
Oct 30 00:51:34.408740 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.408697    6099 cpu_manager.go:414] "containerMap: removing container" podUID="a97377e3-1b2f-4f3e-be46-b6511041b68c" containerName="revoke-keys"
Oct 30 00:51:34.408740 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.408740    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97377e3-1b2f-4f3e-be46-b6511041b68c" containerName="revoke-keys"
Oct 30 00:51:34.408945 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.408909    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97377e3-1b2f-4f3e-be46-b6511041b68c" containerName="revoke-keys"
Oct 30 00:51:34.412516 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.412497    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.432357 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.432331    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"]
Oct 30 00:51:34.514186 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.514152    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.514351 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.514203    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.514351 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.514285    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.514420 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.514348    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc2z6\" (UniqueName: \"kubernetes.io/projected/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-kube-api-access-hc2z6\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.514451 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.514415    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.514621 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.514606    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.514654 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.514641    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.514691 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.514658    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.514752 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.514722    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.616347 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.616313    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.616537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.616360    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.616537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.616386    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.616537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.616414    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.616537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.616488    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.616537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.616519    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.616832 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.616541    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.616832 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.616617    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc2z6\" (UniqueName: \"kubernetes.io/projected/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-kube-api-access-hc2z6\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.616832 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.616653    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.616984 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.616884    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.617036 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.617003    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.617192 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.617157    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.617276 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.617168    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.617496 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.617474    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.618923 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.618901    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.619129 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.619113    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.639471 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.639434    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.639609 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.639590    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc2z6\" (UniqueName: \"kubernetes.io/projected/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-kube-api-access-hc2z6\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:34.726432 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:34.726352    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:35.057984 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:51:35.057953    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee91a08_3dfc_47e1_bac6_bbac9da01bc0.slice/crio-b87a468dbb799b4edc002b52bb6dc538432bfb003f41f73c1982a9fa324678e5 WatchSource:0}: Error finding container b87a468dbb799b4edc002b52bb6dc538432bfb003f41f73c1982a9fa324678e5: Status 404 returned error can't find the container with id b87a468dbb799b4edc002b52bb6dc538432bfb003f41f73c1982a9fa324678e5
Oct 30 00:51:35.060263 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:35.060239    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"]
Oct 30 00:51:35.060585 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:35.060549    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:51:35.060653 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:35.060629    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:51:35.060737 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:35.060677    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:51:35.138176 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:35.138145    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt" event={"ID":"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0","Type":"ContainerStarted","Data":"f8794b2694b56f565efe6dbc0dffa0c03d80ba4c86fcd84e2ebaf599adc5eab3"}
Oct 30 00:51:35.138521 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:35.138205    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt" event={"ID":"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0","Type":"ContainerStarted","Data":"b87a468dbb799b4edc002b52bb6dc538432bfb003f41f73c1982a9fa324678e5"}
Oct 30 00:51:35.307977 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:35.307918    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498"
Oct 30 00:51:35.349756 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:35.349675    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt" podStartSLOduration=1.349660767 podStartE2EDuration="1.349660767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:35.189011937 +0000 UTC m=+989.045919704" watchObservedRunningTime="2026-10-30 00:51:35.349660767 +0000 UTC m=+989.206568522"
Oct 30 00:51:35.429286 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:35.429253    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f50d987e-97c1-4490-afd7-aa57f1b8561f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f50d987e-97c1-4490-afd7-aa57f1b8561f-maas-api-xks-ca\") pod \"f50d987e-97c1-4490-afd7-aa57f1b8561f\" (UID: \"f50d987e-97c1-4490-afd7-aa57f1b8561f\") "
Oct 30 00:51:35.429448 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:35.429435    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f50d987e-97c1-4490-afd7-aa57f1b8561f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f50d987e-97c1-4490-afd7-aa57f1b8561f-maas-api-openshift-ca\") pod \"f50d987e-97c1-4490-afd7-aa57f1b8561f\" (UID: \"f50d987e-97c1-4490-afd7-aa57f1b8561f\") "
Oct 30 00:51:35.429518 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:35.429507    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f50d987e-97c1-4490-afd7-aa57f1b8561f-kube-api-access-gbljp\" (UniqueName: \"kubernetes.io/projected/f50d987e-97c1-4490-afd7-aa57f1b8561f-kube-api-access-gbljp\") pod \"f50d987e-97c1-4490-afd7-aa57f1b8561f\" (UID: \"f50d987e-97c1-4490-afd7-aa57f1b8561f\") "
Oct 30 00:51:35.429793 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:35.429765    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f50d987e-97c1-4490-afd7-aa57f1b8561f-maas-api-openshift-ca" pod "f50d987e-97c1-4490-afd7-aa57f1b8561f" (UID: "f50d987e-97c1-4490-afd7-aa57f1b8561f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:51:35.430099 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:35.430082    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f50d987e-97c1-4490-afd7-aa57f1b8561f-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:35.431444 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:35.431418    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f50d987e-97c1-4490-afd7-aa57f1b8561f-maas-api-xks-ca" pod "f50d987e-97c1-4490-afd7-aa57f1b8561f" (UID: "f50d987e-97c1-4490-afd7-aa57f1b8561f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:51:35.431552 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:35.431499    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f50d987e-97c1-4490-afd7-aa57f1b8561f-kube-api-access-gbljp" pod "f50d987e-97c1-4490-afd7-aa57f1b8561f" (UID: "f50d987e-97c1-4490-afd7-aa57f1b8561f"). InnerVolumeSpecName "kube-api-access-gbljp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:35.532089 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:35.532056    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbljp\" (UniqueName: \"kubernetes.io/projected/f50d987e-97c1-4490-afd7-aa57f1b8561f-kube-api-access-gbljp\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:35.532089 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:35.532086    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f50d987e-97c1-4490-afd7-aa57f1b8561f-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:35.727072 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:35.726967    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:36.147964 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:36.147936    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498"
Oct 30 00:51:36.148320 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:36.147964    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498" event={"ID":"f50d987e-97c1-4490-afd7-aa57f1b8561f","Type":"ContainerDied","Data":"782ffcf4e35045f323141bde7bc81e13bf5966a4fad9f4ce0e2283fa1a1df786"}
Oct 30 00:51:36.148320 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:36.148007    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="782ffcf4e35045f323141bde7bc81e13bf5966a4fad9f4ce0e2283fa1a1df786"
Oct 30 00:51:36.727928 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:36.727897    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 00:51:36.728103 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:36.727949    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt" podUID="6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 00:51:37.727547 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:37.727511    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 00:51:37.727964 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:37.727578    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt" podUID="6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 00:51:38.726673 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:38.726641    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 00:51:38.726904 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:38.726691    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt" podUID="6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 00:51:39.288729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:39.265459    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:51:39.727266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:39.727233    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 00:51:39.727442 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:39.727296    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt" podUID="6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 00:51:40.727877 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:40.727840    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 00:51:40.728583 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:40.728548    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt" podUID="6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 00:51:41.730277 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:41.730239    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 00:51:41.730955 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:41.730923    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt" podUID="6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 00:51:41.775906 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:41.775849    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:41.868027 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:41.867298    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 00:51:41.885362 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:41.885316    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:41.926730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:41.925552    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:51:41.952509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:41.951771    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:41.984544 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:41.984449    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:42.020543 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:42.018846    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded"
Oct 30 00:51:42.469893 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:42.469831    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:42.484731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:42.471469    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:51:42.720787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:42.716819    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:51:42.728736 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:42.727019    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 00:51:42.728736 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:42.727081    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt" podUID="6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 00:51:42.754740 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:42.750797    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:42.779541 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:42.779496    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:42.780075 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:42.780051    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:51:42.894759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:42.891242    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:42.980091 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:42.951456    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:51:42.980091 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:42.952074    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:42.980091 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:42.954167    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:51:42.980382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:42.980180    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:51:42.989776 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:42.980596    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:51:43.012165 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.012118    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:43.030087 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.016514    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:43.030406 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.030381    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:51:43.031143 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.031114    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Oct 30 00:51:43.555752 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.553800    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:43.555752 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.554948    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 00:51:43.582735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.579748    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:51:43.582735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.579822    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:51:43.723862 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.723801    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:43.758732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.731135    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:51:43.760135 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.744517    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 00:51:43.760343 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.760317    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt" podUID="6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 00:51:43.760489 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.754358    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:43.762014 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.761992    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:51:43.850785 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.792648    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 00:51:43.896731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.822806    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:51:43.897610 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.897044    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:43.960494 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.960352    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:43.960494 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.960452    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 00:51:43.989680 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:43.989637    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:44.033936 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:44.032828    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:44.167555 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:44.141916    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd" podUID="96e60f3b-c78f-4de5-aefe-d4e8ee216a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:44.373242 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:44.370881    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:51:44.373242 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:44.370951    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp" podUID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:44.554972 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:44.554925    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:51:44.555340 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:44.555312    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 00:51:44.724967 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:44.724920    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:51:44.748787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:44.740155    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:51:44.817760 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:44.817292    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s" podUID="f5d7156f-8120-4ab7-83ae-f7ab80171e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:44.817760 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:44.775564    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:44.841964 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:44.741370    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 00:51:44.841964 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:44.817803    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt" podUID="6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 00:51:44.841964 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:44.817044    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:51:44.841964 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:44.817853    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl" podUID="90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:44.886464 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:44.886306    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:45.259787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:45.179244    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 00:51:45.259787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:45.183444    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:51:45.259787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:45.221233    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:45.259787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:45.221581    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:45.259787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:45.221864    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:45.259787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:45.222108    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 00:51:45.259787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:45.250835    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": context deadline exceeded"
Oct 30 00:51:45.354732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:45.353834    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:51:45.354732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:45.353894    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh" podUID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:45.585731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:45.585360    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:45.594744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:45.586987    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:51:45.752769 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:45.750281    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:45.853778 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:45.845622    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:45.968741 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:45.895377    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 00:51:46.007798 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.007755    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"]
Oct 30 00:51:46.083730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.031198    6099 cpu_manager.go:414] "containerMap: removing container" podUID="f50d987e-97c1-4490-afd7-aa57f1b8561f" containerName="revoke-keys"
Oct 30 00:51:46.105821 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.099019    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f50d987e-97c1-4490-afd7-aa57f1b8561f" containerName="revoke-keys"
Oct 30 00:51:46.105821 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.099310    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f50d987e-97c1-4490-afd7-aa57f1b8561f" containerName="revoke-keys"
Oct 30 00:51:46.182731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.172737    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"]
Oct 30 00:51:46.186789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.172857    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:46.200735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.179905    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:46.221576 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.221535    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:51:46.245958 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.224594    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 00:51:46.246896 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.224642    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 00:51:46.247654 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.224661    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:51:46.400788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.276082    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5062f6\"/\"distinct-2-w6-5062f6-kserve-self-signed-certs\""
Oct 30 00:51:46.504846 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.495535    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-home\") pod \"distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:46.504846 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.495596    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-tmp-dir\") pod \"distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:46.509763 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.507056    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-model-cache\") pod \"distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:46.509763 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.507132    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6flc\" (UniqueName: \"kubernetes.io/projected/2967eaae-3210-4602-8b3f-f64ff80919cd-kube-api-access-k6flc\") pod \"distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:46.509763 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.507298    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-dshm\") pod \"distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:46.509763 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.507418    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2967eaae-3210-4602-8b3f-f64ff80919cd-tls-certs\") pod \"distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:46.588794 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.587531    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 00:51:46.606730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.598985    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:51:46.664959 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.664856    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-model-cache\") pod \"distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:46.665272 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.665249    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6flc\" (UniqueName: \"kubernetes.io/projected/2967eaae-3210-4602-8b3f-f64ff80919cd-kube-api-access-k6flc\") pod \"distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:46.665613 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.665583    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-dshm\") pod \"distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:46.665923 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.665903    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2967eaae-3210-4602-8b3f-f64ff80919cd-tls-certs\") pod \"distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:46.666313 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.666291    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-home\") pod \"distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:46.666488 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.666470    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-tmp-dir\") pod \"distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:46.668102 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.668080    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-tmp-dir\") pod \"distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:46.668679 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.668660    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-model-cache\") pod \"distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:46.755735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.755667    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:46.786665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.786632    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2967eaae-3210-4602-8b3f-f64ff80919cd-tls-certs\") pod \"distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:46.787210 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.787173    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-dshm\") pod \"distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:46.807278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.807217    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:51:46.807573 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.807540    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt" podUID="6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:46.843130 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.843095    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-home\") pod \"distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:46.883736 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.864805    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:47.273038 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:46.921565    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:47.292430 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.001437    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6flc\" (UniqueName: \"kubernetes.io/projected/2967eaae-3210-4602-8b3f-f64ff80919cd-kube-api-access-k6flc\") pod \"distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") " pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:47.292430 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.052057    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:51:47.292430 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.240782    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:47.292430 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.248360    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 00:51:47.292430 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.248406    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:51:47.293444 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.273320    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:47.303099 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:51:47.303067    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:51:47.309447 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.307586    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:47.366724 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.356043    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:51:47.366724 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.356112    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:51:47.378726 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.368284    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:51:47.394993 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.379838    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:51:47.395271 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.395249    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:51:47.395354 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.395330    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:47.395406 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.395378    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:51:47.395453 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.395409    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:51:47.395453 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.395439    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:51:47.407023 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.400391    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:51:47.407023 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.401075    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:47.407023 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.402885    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:51:47.783324 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:47.783291    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"]
Oct 30 00:51:48.412812 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:48.412776    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" event={"ID":"2967eaae-3210-4602-8b3f-f64ff80919cd","Type":"ContainerStarted","Data":"0f6befec8563c3fe8f865e5889831cf8c4558ba8addff016128fecd0772d8e0c"}
Oct 30 00:51:48.415088 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:48.412825    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" event={"ID":"2967eaae-3210-4602-8b3f-f64ff80919cd","Type":"ContainerStarted","Data":"ea3b9773e243bea1f0a97e8d341abd96a3c7215979aab9fded4ac68e5349689c"}
Oct 30 00:51:48.456916 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:48.456868    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podStartSLOduration=3.456854312 podStartE2EDuration="3.456854312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:48.45632575 +0000 UTC m=+1002.313233507" watchObservedRunningTime="2026-10-30 00:51:48.456854312 +0000 UTC m=+1002.313762123"
Oct 30 00:51:48.880950 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:48.880910    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4"]
Oct 30 00:51:48.888843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:48.888814    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4"
Oct 30 00:51:48.901974 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:48.901945    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4"]
Oct 30 00:51:48.903603 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:48.903567    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:51:48.908920 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:48.908895    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:51:48.909168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:48.908920    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:51:48.961776 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:48.961743    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtxbt\" (UniqueName: \"kubernetes.io/projected/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-kube-api-access-rtxbt\") pod \"maas-api-revoke-sub-7ff1f149-zv6t4\" (UID: \"5c3dff8e-63ac-43bc-be66-c53b21a1ceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4"
Oct 30 00:51:48.961908 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:48.961877    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ff1f149-zv6t4\" (UID: \"5c3dff8e-63ac-43bc-be66-c53b21a1ceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4"
Oct 30 00:51:48.961960 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:48.961945    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ff1f149-zv6t4\" (UID: \"5c3dff8e-63ac-43bc-be66-c53b21a1ceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4"
Oct 30 00:51:49.003827 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:49.003796    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:51:49.063843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:49.063814    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtxbt\" (UniqueName: \"kubernetes.io/projected/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-kube-api-access-rtxbt\") pod \"maas-api-revoke-sub-7ff1f149-zv6t4\" (UID: \"5c3dff8e-63ac-43bc-be66-c53b21a1ceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4"
Oct 30 00:51:49.063970 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:49.063895    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ff1f149-zv6t4\" (UID: \"5c3dff8e-63ac-43bc-be66-c53b21a1ceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4"
Oct 30 00:51:49.064011 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:49.063983    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ff1f149-zv6t4\" (UID: \"5c3dff8e-63ac-43bc-be66-c53b21a1ceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4"
Oct 30 00:51:49.064536 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:49.064508    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ff1f149-zv6t4\" (UID: \"5c3dff8e-63ac-43bc-be66-c53b21a1ceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4"
Oct 30 00:51:49.066295 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:49.066271    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ff1f149-zv6t4\" (UID: \"5c3dff8e-63ac-43bc-be66-c53b21a1ceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4"
Oct 30 00:51:49.086358 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:49.086339    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtxbt\" (UniqueName: \"kubernetes.io/projected/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-kube-api-access-rtxbt\") pod \"maas-api-revoke-sub-7ff1f149-zv6t4\" (UID: \"5c3dff8e-63ac-43bc-be66-c53b21a1ceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4"
Oct 30 00:51:49.211005 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:49.210925    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4"
Oct 30 00:51:49.315739 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:49.315676    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:49.523643 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:51:49.523614    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3dff8e_63ac_43bc_be66_c53b21a1ceb2.slice/crio-a61fb94c70e305b8c115c260e9aa4b000c67791be584e7a4acaf7afc36e8b92b WatchSource:0}: Error finding container a61fb94c70e305b8c115c260e9aa4b000c67791be584e7a4acaf7afc36e8b92b: Status 404 returned error can't find the container with id a61fb94c70e305b8c115c260e9aa4b000c67791be584e7a4acaf7afc36e8b92b
Oct 30 00:51:49.525748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:49.525634    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4"]
Oct 30 00:51:50.432001 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:50.431964    6099 generic.go:356] "Generic (PLEG): container finished" podID="5c3dff8e-63ac-43bc-be66-c53b21a1ceb2" containerID="e43a1feb082b8cf02eb76061fc1cfb02ea1f1cb362c1851d0c8cc0a1ac801525" exitCode=0
Oct 30 00:51:50.432173 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:50.432044    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4" event={"ID":"5c3dff8e-63ac-43bc-be66-c53b21a1ceb2","Type":"ContainerDied","Data":"e43a1feb082b8cf02eb76061fc1cfb02ea1f1cb362c1851d0c8cc0a1ac801525"}
Oct 30 00:51:50.432173 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:50.432098    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4" event={"ID":"5c3dff8e-63ac-43bc-be66-c53b21a1ceb2","Type":"ContainerStarted","Data":"a61fb94c70e305b8c115c260e9aa4b000c67791be584e7a4acaf7afc36e8b92b"}
Oct 30 00:51:51.610748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:51.610722    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4"
Oct 30 00:51:51.718131 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:51.718047    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-maas-api-xks-ca\") pod \"5c3dff8e-63ac-43bc-be66-c53b21a1ceb2\" (UID: \"5c3dff8e-63ac-43bc-be66-c53b21a1ceb2\") "
Oct 30 00:51:51.718403 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:51.718379    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-kube-api-access-rtxbt\" (UniqueName: \"kubernetes.io/projected/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-kube-api-access-rtxbt\") pod \"5c3dff8e-63ac-43bc-be66-c53b21a1ceb2\" (UID: \"5c3dff8e-63ac-43bc-be66-c53b21a1ceb2\") "
Oct 30 00:51:51.718544 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:51.718526    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-maas-api-openshift-ca\") pod \"5c3dff8e-63ac-43bc-be66-c53b21a1ceb2\" (UID: \"5c3dff8e-63ac-43bc-be66-c53b21a1ceb2\") "
Oct 30 00:51:51.718975 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:51.718944    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-maas-api-openshift-ca" pod "5c3dff8e-63ac-43bc-be66-c53b21a1ceb2" (UID: "5c3dff8e-63ac-43bc-be66-c53b21a1ceb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:51:51.719433 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:51.719411    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:51.721794 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:51.721680    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-kube-api-access-rtxbt" pod "5c3dff8e-63ac-43bc-be66-c53b21a1ceb2" (UID: "5c3dff8e-63ac-43bc-be66-c53b21a1ceb2"). InnerVolumeSpecName "kube-api-access-rtxbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:51.723694 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:51.723664    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-maas-api-xks-ca" pod "5c3dff8e-63ac-43bc-be66-c53b21a1ceb2" (UID: "5c3dff8e-63ac-43bc-be66-c53b21a1ceb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:51:51.822135 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:51.822094    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtxbt\" (UniqueName: \"kubernetes.io/projected/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-kube-api-access-rtxbt\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:51.822279 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:51.822185    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:52.138639 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.138608    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j"]
Oct 30 00:51:52.139349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.139335    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5c3dff8e-63ac-43bc-be66-c53b21a1ceb2" containerName="revoke-keys"
Oct 30 00:51:52.139399 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.139352    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c3dff8e-63ac-43bc-be66-c53b21a1ceb2" containerName="revoke-keys"
Oct 30 00:51:52.139479 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.139471    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c3dff8e-63ac-43bc-be66-c53b21a1ceb2" containerName="revoke-keys"
Oct 30 00:51:52.144536 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.144517    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j"
Oct 30 00:51:52.164177 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.164150    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j"]
Oct 30 00:51:52.228115 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.228082    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed962bb-ac51-4b94-9443-2661169d35fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ce23d71-pjq5j\" (UID: \"7ed962bb-ac51-4b94-9443-2661169d35fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j"
Oct 30 00:51:52.228409 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.228378    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed962bb-ac51-4b94-9443-2661169d35fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ce23d71-pjq5j\" (UID: \"7ed962bb-ac51-4b94-9443-2661169d35fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j"
Oct 30 00:51:52.228587 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.228565    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx7xd\" (UniqueName: \"kubernetes.io/projected/7ed962bb-ac51-4b94-9443-2661169d35fe-kube-api-access-jx7xd\") pod \"maas-api-revoke-sub-3ce23d71-pjq5j\" (UID: \"7ed962bb-ac51-4b94-9443-2661169d35fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j"
Oct 30 00:51:52.329652 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.329622    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed962bb-ac51-4b94-9443-2661169d35fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ce23d71-pjq5j\" (UID: \"7ed962bb-ac51-4b94-9443-2661169d35fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j"
Oct 30 00:51:52.329851 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.329770    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx7xd\" (UniqueName: \"kubernetes.io/projected/7ed962bb-ac51-4b94-9443-2661169d35fe-kube-api-access-jx7xd\") pod \"maas-api-revoke-sub-3ce23d71-pjq5j\" (UID: \"7ed962bb-ac51-4b94-9443-2661169d35fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j"
Oct 30 00:51:52.329851 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.329810    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed962bb-ac51-4b94-9443-2661169d35fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ce23d71-pjq5j\" (UID: \"7ed962bb-ac51-4b94-9443-2661169d35fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j"
Oct 30 00:51:52.330440 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.330411    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed962bb-ac51-4b94-9443-2661169d35fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ce23d71-pjq5j\" (UID: \"7ed962bb-ac51-4b94-9443-2661169d35fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j"
Oct 30 00:51:52.331969 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.331949    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed962bb-ac51-4b94-9443-2661169d35fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ce23d71-pjq5j\" (UID: \"7ed962bb-ac51-4b94-9443-2661169d35fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j"
Oct 30 00:51:52.352281 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.352257    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx7xd\" (UniqueName: \"kubernetes.io/projected/7ed962bb-ac51-4b94-9443-2661169d35fe-kube-api-access-jx7xd\") pod \"maas-api-revoke-sub-3ce23d71-pjq5j\" (UID: \"7ed962bb-ac51-4b94-9443-2661169d35fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j"
Oct 30 00:51:52.453664 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.453575    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4" event={"ID":"5c3dff8e-63ac-43bc-be66-c53b21a1ceb2","Type":"ContainerDied","Data":"a61fb94c70e305b8c115c260e9aa4b000c67791be584e7a4acaf7afc36e8b92b"}
Oct 30 00:51:52.453993 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.453960    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a61fb94c70e305b8c115c260e9aa4b000c67791be584e7a4acaf7afc36e8b92b"
Oct 30 00:51:52.453993 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.453616    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4"
Oct 30 00:51:52.460364 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.460340    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j"
Oct 30 00:51:52.776085 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:51:52.776056    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed962bb_ac51_4b94_9443_2661169d35fe.slice/crio-2d1ed090619af8e5ee7de75bad9606f6eacaee7e48b6489391e77de9ea69b39a WatchSource:0}: Error finding container 2d1ed090619af8e5ee7de75bad9606f6eacaee7e48b6489391e77de9ea69b39a: Status 404 returned error can't find the container with id 2d1ed090619af8e5ee7de75bad9606f6eacaee7e48b6489391e77de9ea69b39a
Oct 30 00:51:52.779179 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:52.779154    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j"]
Oct 30 00:51:53.463645 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:53.463549    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j" event={"ID":"7ed962bb-ac51-4b94-9443-2661169d35fe","Type":"ContainerStarted","Data":"0dbe9857f45f06441cac64c8c7b8ced3ef7af8669444c290b1069f54f3a8281b"}
Oct 30 00:51:53.463824 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:53.463644    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j" event={"ID":"7ed962bb-ac51-4b94-9443-2661169d35fe","Type":"ContainerStarted","Data":"2d1ed090619af8e5ee7de75bad9606f6eacaee7e48b6489391e77de9ea69b39a"}
Oct 30 00:51:53.505313 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:53.505261    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j" podStartSLOduration=1.505248603 podStartE2EDuration="1.505248603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:53.502521459 +0000 UTC m=+1007.359429227" watchObservedRunningTime="2026-10-30 00:51:53.505248603 +0000 UTC m=+1007.362156357"
Oct 30 00:51:54.473208 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:54.473169    6099 generic.go:356] "Generic (PLEG): container finished" podID="7ed962bb-ac51-4b94-9443-2661169d35fe" containerID="0dbe9857f45f06441cac64c8c7b8ced3ef7af8669444c290b1069f54f3a8281b" exitCode=0
Oct 30 00:51:54.473624 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:54.473291    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j" event={"ID":"7ed962bb-ac51-4b94-9443-2661169d35fe","Type":"ContainerDied","Data":"0dbe9857f45f06441cac64c8c7b8ced3ef7af8669444c290b1069f54f3a8281b"}
Oct 30 00:51:55.667690 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:55.667658    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j"
Oct 30 00:51:55.786139 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:55.786059    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ed962bb-ac51-4b94-9443-2661169d35fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed962bb-ac51-4b94-9443-2661169d35fe-maas-api-xks-ca\") pod \"7ed962bb-ac51-4b94-9443-2661169d35fe\" (UID: \"7ed962bb-ac51-4b94-9443-2661169d35fe\") "
Oct 30 00:51:55.786320 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:55.786304    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed962bb-ac51-4b94-9443-2661169d35fe-kube-api-access-jx7xd\" (UniqueName: \"kubernetes.io/projected/7ed962bb-ac51-4b94-9443-2661169d35fe-kube-api-access-jx7xd\") pod \"7ed962bb-ac51-4b94-9443-2661169d35fe\" (UID: \"7ed962bb-ac51-4b94-9443-2661169d35fe\") "
Oct 30 00:51:55.786420 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:55.786405    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed962bb-ac51-4b94-9443-2661169d35fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed962bb-ac51-4b94-9443-2661169d35fe-maas-api-openshift-ca\") pod \"7ed962bb-ac51-4b94-9443-2661169d35fe\" (UID: \"7ed962bb-ac51-4b94-9443-2661169d35fe\") "
Oct 30 00:51:55.786810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:55.786784    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed962bb-ac51-4b94-9443-2661169d35fe-maas-api-openshift-ca" pod "7ed962bb-ac51-4b94-9443-2661169d35fe" (UID: "7ed962bb-ac51-4b94-9443-2661169d35fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:51:55.787193 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:55.787175    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed962bb-ac51-4b94-9443-2661169d35fe-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:55.788311 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:55.788288    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed962bb-ac51-4b94-9443-2661169d35fe-maas-api-xks-ca" pod "7ed962bb-ac51-4b94-9443-2661169d35fe" (UID: "7ed962bb-ac51-4b94-9443-2661169d35fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:51:55.788433 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:55.788411    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed962bb-ac51-4b94-9443-2661169d35fe-kube-api-access-jx7xd" pod "7ed962bb-ac51-4b94-9443-2661169d35fe" (UID: "7ed962bb-ac51-4b94-9443-2661169d35fe"). InnerVolumeSpecName "kube-api-access-jx7xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:55.888819 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:55.888791    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx7xd\" (UniqueName: \"kubernetes.io/projected/7ed962bb-ac51-4b94-9443-2661169d35fe-kube-api-access-jx7xd\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:55.888819 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:55.888816    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed962bb-ac51-4b94-9443-2661169d35fe-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:51:56.491154 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:56.491128    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j"
Oct 30 00:51:56.491154 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:56.491136    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j" event={"ID":"7ed962bb-ac51-4b94-9443-2661169d35fe","Type":"ContainerDied","Data":"2d1ed090619af8e5ee7de75bad9606f6eacaee7e48b6489391e77de9ea69b39a"}
Oct 30 00:51:56.491351 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:56.491204    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d1ed090619af8e5ee7de75bad9606f6eacaee7e48b6489391e77de9ea69b39a"
Oct 30 00:51:57.316364 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:57.316331    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:57.328650 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:57.328624    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:57.510323 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:57.510289    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:51:57.739531 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:57.739456    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj"]
Oct 30 00:51:57.740149 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:57.740136    6099 cpu_manager.go:414] "containerMap: removing container" podUID="7ed962bb-ac51-4b94-9443-2661169d35fe" containerName="revoke-keys"
Oct 30 00:51:57.740201 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:57.740151    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed962bb-ac51-4b94-9443-2661169d35fe" containerName="revoke-keys"
Oct 30 00:51:57.740279 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:57.740270    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed962bb-ac51-4b94-9443-2661169d35fe" containerName="revoke-keys"
Oct 30 00:51:57.744437 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:57.744412    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj"
Oct 30 00:51:57.758965 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:57.758941    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:51:57.759112 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:57.759027    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:51:57.759112 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:57.759100    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:51:57.759219 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:57.759137    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:51:57.763385 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:57.763355    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj"]
Oct 30 00:51:57.919136 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:57.919104    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fbc0473-def5-4b5b-b78b-7719d7a6e443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38441b21-g29wj\" (UID: \"6fbc0473-def5-4b5b-b78b-7719d7a6e443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj"
Oct 30 00:51:57.919297 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:57.919260    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k2nj\" (UniqueName: \"kubernetes.io/projected/6fbc0473-def5-4b5b-b78b-7719d7a6e443-kube-api-access-9k2nj\") pod \"maas-api-revoke-sub-38441b21-g29wj\" (UID: \"6fbc0473-def5-4b5b-b78b-7719d7a6e443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj"
Oct 30 00:51:57.919349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:57.919320    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fbc0473-def5-4b5b-b78b-7719d7a6e443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38441b21-g29wj\" (UID: \"6fbc0473-def5-4b5b-b78b-7719d7a6e443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj"
Oct 30 00:51:58.020515 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:58.020441    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fbc0473-def5-4b5b-b78b-7719d7a6e443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38441b21-g29wj\" (UID: \"6fbc0473-def5-4b5b-b78b-7719d7a6e443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj"
Oct 30 00:51:58.020658 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:58.020596    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k2nj\" (UniqueName: \"kubernetes.io/projected/6fbc0473-def5-4b5b-b78b-7719d7a6e443-kube-api-access-9k2nj\") pod \"maas-api-revoke-sub-38441b21-g29wj\" (UID: \"6fbc0473-def5-4b5b-b78b-7719d7a6e443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj"
Oct 30 00:51:58.020764 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:58.020691    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fbc0473-def5-4b5b-b78b-7719d7a6e443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38441b21-g29wj\" (UID: \"6fbc0473-def5-4b5b-b78b-7719d7a6e443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj"
Oct 30 00:51:58.021179 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:58.021160    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fbc0473-def5-4b5b-b78b-7719d7a6e443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38441b21-g29wj\" (UID: \"6fbc0473-def5-4b5b-b78b-7719d7a6e443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj"
Oct 30 00:51:58.022847 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:58.022828    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fbc0473-def5-4b5b-b78b-7719d7a6e443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38441b21-g29wj\" (UID: \"6fbc0473-def5-4b5b-b78b-7719d7a6e443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj"
Oct 30 00:51:58.043285 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:58.043261    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k2nj\" (UniqueName: \"kubernetes.io/projected/6fbc0473-def5-4b5b-b78b-7719d7a6e443-kube-api-access-9k2nj\") pod \"maas-api-revoke-sub-38441b21-g29wj\" (UID: \"6fbc0473-def5-4b5b-b78b-7719d7a6e443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj"
Oct 30 00:51:58.061699 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:58.061680    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj"
Oct 30 00:51:58.368039 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:51:58.368006    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fbc0473_def5_4b5b_b78b_7719d7a6e443.slice/crio-708c5e3803681dc13b34abcbd8514d384b3ea6a1f0da1aa991bd6d92614f1d91 WatchSource:0}: Error finding container 708c5e3803681dc13b34abcbd8514d384b3ea6a1f0da1aa991bd6d92614f1d91: Status 404 returned error can't find the container with id 708c5e3803681dc13b34abcbd8514d384b3ea6a1f0da1aa991bd6d92614f1d91
Oct 30 00:51:58.369802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:58.369778    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj"]
Oct 30 00:51:58.507887 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:58.507851    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj" event={"ID":"6fbc0473-def5-4b5b-b78b-7719d7a6e443","Type":"ContainerStarted","Data":"708c5e3803681dc13b34abcbd8514d384b3ea6a1f0da1aa991bd6d92614f1d91"}
Oct 30 00:51:59.518191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:59.518153    6099 generic.go:356] "Generic (PLEG): container finished" podID="6fbc0473-def5-4b5b-b78b-7719d7a6e443" containerID="c1364c264ff91e6148c4557e558f24d4bdae5ab9f84437fd4bc28fb20a733691" exitCode=0
Oct 30 00:51:59.518543 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:51:59.518213    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj" event={"ID":"6fbc0473-def5-4b5b-b78b-7719d7a6e443","Type":"ContainerDied","Data":"c1364c264ff91e6148c4557e558f24d4bdae5ab9f84437fd4bc28fb20a733691"}
Oct 30 00:52:00.686450 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:00.686426    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj"
Oct 30 00:52:00.863817 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:00.863791    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fbc0473-def5-4b5b-b78b-7719d7a6e443-kube-api-access-9k2nj\" (UniqueName: \"kubernetes.io/projected/6fbc0473-def5-4b5b-b78b-7719d7a6e443-kube-api-access-9k2nj\") pod \"6fbc0473-def5-4b5b-b78b-7719d7a6e443\" (UID: \"6fbc0473-def5-4b5b-b78b-7719d7a6e443\") "
Oct 30 00:52:00.864005 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:00.863848    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fbc0473-def5-4b5b-b78b-7719d7a6e443-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fbc0473-def5-4b5b-b78b-7719d7a6e443-maas-api-openshift-ca\") pod \"6fbc0473-def5-4b5b-b78b-7719d7a6e443\" (UID: \"6fbc0473-def5-4b5b-b78b-7719d7a6e443\") "
Oct 30 00:52:00.864005 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:00.863888    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fbc0473-def5-4b5b-b78b-7719d7a6e443-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fbc0473-def5-4b5b-b78b-7719d7a6e443-maas-api-xks-ca\") pod \"6fbc0473-def5-4b5b-b78b-7719d7a6e443\" (UID: \"6fbc0473-def5-4b5b-b78b-7719d7a6e443\") "
Oct 30 00:52:00.864263 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:00.864238    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fbc0473-def5-4b5b-b78b-7719d7a6e443-maas-api-openshift-ca" pod "6fbc0473-def5-4b5b-b78b-7719d7a6e443" (UID: "6fbc0473-def5-4b5b-b78b-7719d7a6e443"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:52:00.864447 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:00.864433    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fbc0473-def5-4b5b-b78b-7719d7a6e443-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:00.865892 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:00.865865    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fbc0473-def5-4b5b-b78b-7719d7a6e443-maas-api-xks-ca" pod "6fbc0473-def5-4b5b-b78b-7719d7a6e443" (UID: "6fbc0473-def5-4b5b-b78b-7719d7a6e443"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:52:00.865951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:00.865939    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fbc0473-def5-4b5b-b78b-7719d7a6e443-kube-api-access-9k2nj" pod "6fbc0473-def5-4b5b-b78b-7719d7a6e443" (UID: "6fbc0473-def5-4b5b-b78b-7719d7a6e443"). InnerVolumeSpecName "kube-api-access-9k2nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:52:00.966403 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:00.966356    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k2nj\" (UniqueName: \"kubernetes.io/projected/6fbc0473-def5-4b5b-b78b-7719d7a6e443-kube-api-access-9k2nj\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:00.966403 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:00.966396    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fbc0473-def5-4b5b-b78b-7719d7a6e443-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:01.536421 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:01.536387    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj"
Oct 30 00:52:01.536622 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:01.536422    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj" event={"ID":"6fbc0473-def5-4b5b-b78b-7719d7a6e443","Type":"ContainerDied","Data":"708c5e3803681dc13b34abcbd8514d384b3ea6a1f0da1aa991bd6d92614f1d91"}
Oct 30 00:52:01.536622 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:01.536469    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="708c5e3803681dc13b34abcbd8514d384b3ea6a1f0da1aa991bd6d92614f1d91"
Oct 30 00:52:03.610511 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.610478    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"]
Oct 30 00:52:03.611216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.611199    6099 cpu_manager.go:414] "containerMap: removing container" podUID="6fbc0473-def5-4b5b-b78b-7719d7a6e443" containerName="revoke-keys"
Oct 30 00:52:03.611216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.611217    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fbc0473-def5-4b5b-b78b-7719d7a6e443" containerName="revoke-keys"
Oct 30 00:52:03.611334 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.611325    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fbc0473-def5-4b5b-b78b-7719d7a6e443" containerName="revoke-keys"
Oct 30 00:52:03.615338 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.615307    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.643900 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.643859    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"]
Oct 30 00:52:03.706056 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.706022    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.706217 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.706082    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.706217 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.706180    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.706326 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.706220    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.706326 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.706243    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.706326 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.706301    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpjht\" (UniqueName: \"kubernetes.io/projected/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-kube-api-access-kpjht\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.706436 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.706340    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.706469 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.706455    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.706570 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.706551    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.807733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.807679    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.807876 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.807801    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.807876 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.807854    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.807982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.807889    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.807982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.807920    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.808100 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.807978    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.808100 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.808035    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.808100 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.808038    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.808100 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.808079    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpjht\" (UniqueName: \"kubernetes.io/projected/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-kube-api-access-kpjht\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.808289 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.808119    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.808289 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.808226    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.808395 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.808315    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.808447 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.808421    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.808753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.808726    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.810404 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.810368    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.810519 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.810457    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.830284 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.830261    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.830701 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.830684    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpjht\" (UniqueName: \"kubernetes.io/projected/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-kube-api-access-kpjht\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:03.930400 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:03.930327    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:04.256110 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:52:04.256075    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb3ac3a_8a32_4a49_98c3_15f3cee9f391.slice/crio-b2c43613a6482b81497b8337e44b5087f88e74f9fa7df9c8e311a1c52c1c85da WatchSource:0}: Error finding container b2c43613a6482b81497b8337e44b5087f88e74f9fa7df9c8e311a1c52c1c85da: Status 404 returned error can't find the container with id b2c43613a6482b81497b8337e44b5087f88e74f9fa7df9c8e311a1c52c1c85da
Oct 30 00:52:04.258844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:04.258800    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:52:04.258939 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:04.258874    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:52:04.258939 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:04.258903    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:52:04.263003 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:04.262979    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"]
Oct 30 00:52:04.567162 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:04.567108    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f" event={"ID":"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391","Type":"ContainerStarted","Data":"3c9bc3a0c17ba83770c3f71845a248977550d879d8ec56b14d14232b4ded035c"}
Oct 30 00:52:04.567422 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:04.567206    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f" event={"ID":"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391","Type":"ContainerStarted","Data":"b2c43613a6482b81497b8337e44b5087f88e74f9fa7df9c8e311a1c52c1c85da"}
Oct 30 00:52:04.614823 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:04.614770    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f" podStartSLOduration=1.6147547850000001 podStartE2EDuration="1.614754785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:04.614257398 +0000 UTC m=+1018.471165145" watchObservedRunningTime="2026-10-30 00:52:04.614754785 +0000 UTC m=+1018.471662539"
Oct 30 00:52:04.930600 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:04.930515    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:05.955731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:05.955314    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 00:52:05.955731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:05.955428    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f" podUID="4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 00:52:06.874581 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:06.867121    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:07.004239 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.004201    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 00:52:07.022441 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.022396    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f" podUID="4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 00:52:07.210789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.209926    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:52:07.212875 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.212810    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:07.213088 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.212973    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:52:07.213270 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.213000    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded"
Oct 30 00:52:07.245116 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.245048    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:07.245504 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.245463    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 00:52:07.322741 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.314341    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg"]
Oct 30 00:52:07.413752 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.402085    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg"]
Oct 30 00:52:07.413752 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.402146    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"]
Oct 30 00:52:07.416110 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.416084    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg"
Oct 30 00:52:07.454733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.448086    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:52:07.472784 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.469467    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94gkd\" (UniqueName: \"kubernetes.io/projected/740d18f1-010f-4859-96a3-c65d1dba7abf-kube-api-access-94gkd\") pod \"maas-api-revoke-sub-64f60dc1-ckpzg\" (UID: \"740d18f1-010f-4859-96a3-c65d1dba7abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg"
Oct 30 00:52:07.472784 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.469543    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740d18f1-010f-4859-96a3-c65d1dba7abf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64f60dc1-ckpzg\" (UID: \"740d18f1-010f-4859-96a3-c65d1dba7abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg"
Oct 30 00:52:07.472784 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.469674    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740d18f1-010f-4859-96a3-c65d1dba7abf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64f60dc1-ckpzg\" (UID: \"740d18f1-010f-4859-96a3-c65d1dba7abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg"
Oct 30 00:52:07.502732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.499863    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:52:07.502732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.499951    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:52:07.502732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.500099    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:52:07.528423 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.527677    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": context deadline exceeded"
Oct 30 00:52:07.607236 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.607190    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:07.625743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.621602    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740d18f1-010f-4859-96a3-c65d1dba7abf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64f60dc1-ckpzg\" (UID: \"740d18f1-010f-4859-96a3-c65d1dba7abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg"
Oct 30 00:52:07.625743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.621827    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740d18f1-010f-4859-96a3-c65d1dba7abf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64f60dc1-ckpzg\" (UID: \"740d18f1-010f-4859-96a3-c65d1dba7abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg"
Oct 30 00:52:07.625743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.622218    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94gkd\" (UniqueName: \"kubernetes.io/projected/740d18f1-010f-4859-96a3-c65d1dba7abf-kube-api-access-94gkd\") pod \"maas-api-revoke-sub-64f60dc1-ckpzg\" (UID: \"740d18f1-010f-4859-96a3-c65d1dba7abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg"
Oct 30 00:52:07.625743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.624207    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740d18f1-010f-4859-96a3-c65d1dba7abf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64f60dc1-ckpzg\" (UID: \"740d18f1-010f-4859-96a3-c65d1dba7abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg"
Oct 30 00:52:07.656243 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.656203    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:07.657553 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.657528    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"]
Oct 30 00:52:07.657820 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.657802    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:07.697175 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.697142    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 00:52:07.704781 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.698278    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 00:52:07.704992 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.698392    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pqwzr\""
Oct 30 00:52:07.705114 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.698434    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 00:52:07.844226 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.844195    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740d18f1-010f-4859-96a3-c65d1dba7abf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64f60dc1-ckpzg\" (UID: \"740d18f1-010f-4859-96a3-c65d1dba7abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg"
Oct 30 00:52:07.844488 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.844458    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:07.891730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.888452    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94gkd\" (UniqueName: \"kubernetes.io/projected/740d18f1-010f-4859-96a3-c65d1dba7abf-kube-api-access-94gkd\") pod \"maas-api-revoke-sub-64f60dc1-ckpzg\" (UID: \"740d18f1-010f-4859-96a3-c65d1dba7abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg"
Oct 30 00:52:07.891730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.891559    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:07.892366 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.892337    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:07.901881 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.901842    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:07.902319 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.892721    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:52:07.985475 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.985443    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg"
Oct 30 00:52:07.994268 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:07.994231    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/423014b8-4236-4135-8410-b53876b9a755-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:08.179610 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.179576    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:08.201693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.201658    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:08.202116 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.202089    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:08.245732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.027137    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 00:52:08.264825 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.264761    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f" podUID="4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 00:52:08.367920 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.367870    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:08.507877 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.393792    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:52:08.566424 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.394385    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:08.613768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.396507    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:08.613768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.415285    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:52:08.613768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.415322    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 00:52:08.613768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.440919    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/423014b8-4236-4135-8410-b53876b9a755-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:08.613768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.581432    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg6m8\" (UniqueName: \"kubernetes.io/projected/423014b8-4236-4135-8410-b53876b9a755-kube-api-access-jg6m8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:08.613768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.581529    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423014b8-4236-4135-8410-b53876b9a755-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:08.613768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.554587    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:08.613768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.554624    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:08.648919 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.648856    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:08.680773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.671801    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:08.818550 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.818499    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:08.942731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.914998    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 00:52:09.029127 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.916889    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 00:52:09.040096 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.917736    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:09.080155 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.080070    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/423014b8-4236-4135-8410-b53876b9a755-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:09.100932 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.100895    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg6m8\" (UniqueName: \"kubernetes.io/projected/423014b8-4236-4135-8410-b53876b9a755-kube-api-access-jg6m8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:09.102402 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.102374    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423014b8-4236-4135-8410-b53876b9a755-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:09.110197 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.110167    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/423014b8-4236-4135-8410-b53876b9a755-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:09.132826 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.132792    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:09.141400 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.141363    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:09.158936 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.896681    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:09.160526 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:08.946052    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 00:52:09.160726 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.160673    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f" podUID="4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 00:52:09.160869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.087854    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:52:09.161024 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.161011    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:52:09.161183 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.161169    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:52:09.161332 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.161320    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:52:09.161458 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.161446    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:52:09.161579 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.161568    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:52:09.161721 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.161692    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:52:09.161847 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.161835    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:52:09.174771 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.161961    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:52:09.175103 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.175074    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:52:09.175996 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.118609    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/423014b8-4236-4135-8410-b53876b9a755-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:09.177307 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.038407    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:09.449779 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.449678    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:09.462402 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.462370    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:09.506217 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.485836    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/423014b8-4236-4135-8410-b53876b9a755-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:09.508240 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.508206    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423014b8-4236-4135-8410-b53876b9a755-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:09.911731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.510340    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg6m8\" (UniqueName: \"kubernetes.io/projected/423014b8-4236-4135-8410-b53876b9a755-kube-api-access-jg6m8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhlj6\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:09.911731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.527585    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:09.911731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.894064    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:09.911731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.642264    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:09.911731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.644534    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:52:09.911731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.644658    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 00:52:09.911731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.684135    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:09.911731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.602112    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:52:09.911731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.749294    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:09.958277 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:09.958197    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded"
Oct 30 00:52:10.226162 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:10.226032    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:10.258801 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:10.258761    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 00:52:10.259161 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:10.259118    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f" podUID="4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 00:52:10.261576 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:10.261531    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:10.262752 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:10.262725    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 00:52:10.734012 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:10.733964    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:10.929133 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:10.929079    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:10.929506 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:10.929474    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 00:52:10.929781 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:10.929756    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:10.933729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:10.929962    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:10.933729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:10.930162    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 00:52:10.934088 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:10.934059    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:10.939000 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:10.930002    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 00:52:10.957235 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:10.930292    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": context deadline exceeded"
Oct 30 00:52:11.181861 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.168130    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hknzb" podUID="4f0e2175-e7cb-48f7-b2ac-1c18e1890cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:11.230730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.217171    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:52:11.230730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.217251    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w" podUID="cc360f77-72ad-4729-a4e7-bf542d69b142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:11.231112 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.231067    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:11.283799 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.280930    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 00:52:11.283799 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.281178    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 00:52:11.503598 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.501693    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Oct 30 00:52:11.783763 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.763938    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-5f78687d5b-rcxc5" podUID="1569a420-ef8b-4ca9-86c1-cde78845b4b6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.39:50052\" within 1s: context deadline exceeded"
Oct 30 00:52:11.848793 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.848740    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm" podUID="d94d18a3-84f0-4225-bf91-19ce9a058574" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:11.898676 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.873318    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv" podUID="5c8179b0-96f4-4647-b926-5969b4afecd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:11.966168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.934820    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:52:11.966168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.934884    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f" podUID="4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:11.966168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.935758    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 00:52:11.966168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.936357    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:52:11.966168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.945818    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:11.966168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.946883    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 00:52:11.966168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.950805    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:11.966168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.951055    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:11.966168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.951222    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:52:11.966168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.952988    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 00:52:11.966168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.953025    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f" podUID="4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 00:52:11.966168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:11.958217    6099 trace.go:236] Trace[1077344514]: "Calculate volume metrics of dshm for pod e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" (30-Oct-2026 00:52:10.855) (total time: 1102ms):
Oct 30 00:52:11.966168 ip-10-0-130-245 kubenswrapper[6099]: Trace[1077344514]: [1.102283765s] [1.102283765s] END
Oct 30 00:52:12.023886 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:52:12.000115    6099 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.628s"
Oct 30 00:52:12.023886 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.000189    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:52:12.023886 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.001884    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:12.027447 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:52:12.027413    6099 logging.go:55] [core] [Channel #458 SubChannel #459]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.39:50052", ServerName: "10.134.0.39:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.134.0.2:44888->10.134.0.39:50052: use of closed network connection"
Oct 30 00:52:12.028132 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.028112    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:52:12.034893 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.032566    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:52:12.050760 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.036628    6099 trace.go:236] Trace[1982354706]: "Calculate volume metrics of manifests for pod opendatahub/opendatahub-operator-controller-manager-6f85c8f7c-k29rq" (30-Oct-2026 00:52:10.309) (total time: 1726ms):
Oct 30 00:52:12.050760 ip-10-0-130-245 kubenswrapper[6099]: Trace[1982354706]: [1.726992803s] [1.726992803s] END
Oct 30 00:52:12.050760 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.042873    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:52:12.114855 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.101091    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:52:12.193355 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.192023    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:52:12.263998 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.254870    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:52:12.264255 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.264233    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:52:12.264411 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.264399    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:52:12.318419 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.318393    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:52:12.318625 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.318442    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:52:12.326876 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.326849    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg"]
Oct 30 00:52:12.408018 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.407988    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"]
Oct 30 00:52:12.509180 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.509156    6099 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:52:12.630817 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.630785    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg" event={"ID":"740d18f1-010f-4859-96a3-c65d1dba7abf","Type":"ContainerStarted","Data":"7dcaf47a40011ad9efcfb2219fe7059b5f11fdbcbeea85340116e915ae37d2ae"}
Oct 30 00:52:12.632728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.632682    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6" event={"ID":"423014b8-4236-4135-8410-b53876b9a755","Type":"ContainerStarted","Data":"9009aa363125a0d868bb9902c2b8cfd92efc5bc31d8a7cb9be252939fb7ecad1"}
Oct 30 00:52:12.930729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.930624    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 00:52:12.930729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:12.930672    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f" podUID="4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 00:52:13.645972 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:13.645211    6099 generic.go:356] "Generic (PLEG): container finished" podID="740d18f1-010f-4859-96a3-c65d1dba7abf" containerID="a601bf87e6ede97c4ec08e6c0b074506e21e7aaae97980e77e191bd31334a344" exitCode=0
Oct 30 00:52:13.645972 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:13.645505    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg" event={"ID":"740d18f1-010f-4859-96a3-c65d1dba7abf","Type":"ContainerDied","Data":"a601bf87e6ede97c4ec08e6c0b074506e21e7aaae97980e77e191bd31334a344"}
Oct 30 00:52:13.930780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:13.930742    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 00:52:13.931275 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:13.930804    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f" podUID="4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 00:52:14.863454 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:14.863422    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg"
Oct 30 00:52:14.930859 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:14.930768    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 00:52:14.930859 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:14.930827    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f" podUID="4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 00:52:14.970535 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:14.970495    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/740d18f1-010f-4859-96a3-c65d1dba7abf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740d18f1-010f-4859-96a3-c65d1dba7abf-maas-api-openshift-ca\") pod \"740d18f1-010f-4859-96a3-c65d1dba7abf\" (UID: \"740d18f1-010f-4859-96a3-c65d1dba7abf\") "
Oct 30 00:52:14.970826 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:14.970801    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/740d18f1-010f-4859-96a3-c65d1dba7abf-kube-api-access-94gkd\" (UniqueName: \"kubernetes.io/projected/740d18f1-010f-4859-96a3-c65d1dba7abf-kube-api-access-94gkd\") pod \"740d18f1-010f-4859-96a3-c65d1dba7abf\" (UID: \"740d18f1-010f-4859-96a3-c65d1dba7abf\") "
Oct 30 00:52:14.970945 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:14.970918    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740d18f1-010f-4859-96a3-c65d1dba7abf-maas-api-openshift-ca" pod "740d18f1-010f-4859-96a3-c65d1dba7abf" (UID: "740d18f1-010f-4859-96a3-c65d1dba7abf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:52:14.971011 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:14.970965    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/740d18f1-010f-4859-96a3-c65d1dba7abf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740d18f1-010f-4859-96a3-c65d1dba7abf-maas-api-xks-ca\") pod \"740d18f1-010f-4859-96a3-c65d1dba7abf\" (UID: \"740d18f1-010f-4859-96a3-c65d1dba7abf\") "
Oct 30 00:52:14.971754 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:14.971731    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740d18f1-010f-4859-96a3-c65d1dba7abf-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:14.973686 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:14.973659    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740d18f1-010f-4859-96a3-c65d1dba7abf-kube-api-access-94gkd" pod "740d18f1-010f-4859-96a3-c65d1dba7abf" (UID: "740d18f1-010f-4859-96a3-c65d1dba7abf"). InnerVolumeSpecName "kube-api-access-94gkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:52:14.981255 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:14.981233    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/740d18f1-010f-4859-96a3-c65d1dba7abf-maas-api-xks-ca" pod "740d18f1-010f-4859-96a3-c65d1dba7abf" (UID: "740d18f1-010f-4859-96a3-c65d1dba7abf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:52:15.073893 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:15.073853    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94gkd\" (UniqueName: \"kubernetes.io/projected/740d18f1-010f-4859-96a3-c65d1dba7abf-kube-api-access-94gkd\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:15.073893 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:15.073894    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740d18f1-010f-4859-96a3-c65d1dba7abf-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:15.673262 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:15.673231    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg"
Oct 30 00:52:15.673486 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:15.673345    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg" event={"ID":"740d18f1-010f-4859-96a3-c65d1dba7abf","Type":"ContainerDied","Data":"7dcaf47a40011ad9efcfb2219fe7059b5f11fdbcbeea85340116e915ae37d2ae"}
Oct 30 00:52:15.673486 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:15.673383    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dcaf47a40011ad9efcfb2219fe7059b5f11fdbcbeea85340116e915ae37d2ae"
Oct 30 00:52:15.959713 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:15.959671    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:52:15.960085 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:15.959756    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:52:15.995482 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:15.995449    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:15.996470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:15.996445    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:15.997302 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:15.997278    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:52:16.300614 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.300586    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn"]
Oct 30 00:52:16.301284 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.301268    6099 cpu_manager.go:414] "containerMap: removing container" podUID="740d18f1-010f-4859-96a3-c65d1dba7abf" containerName="revoke-keys"
Oct 30 00:52:16.301284 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.301284    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="740d18f1-010f-4859-96a3-c65d1dba7abf" containerName="revoke-keys"
Oct 30 00:52:16.301432 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.301423    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="740d18f1-010f-4859-96a3-c65d1dba7abf" containerName="revoke-keys"
Oct 30 00:52:16.304997 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.304979    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn"
Oct 30 00:52:16.322758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.322661    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:52:16.322901 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.322797    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:52:16.325982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.325954    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn"]
Oct 30 00:52:16.353757 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.347904    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:52:16.353757 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.347974    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:52:16.400031 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.400003    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533b61f4-794e-48b8-aa70-4dd498211e12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adee9bd3-jshpn\" (UID: \"533b61f4-794e-48b8-aa70-4dd498211e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn"
Oct 30 00:52:16.400154 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.400055    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6fpf\" (UniqueName: \"kubernetes.io/projected/533b61f4-794e-48b8-aa70-4dd498211e12-kube-api-access-g6fpf\") pod \"maas-api-revoke-sub-adee9bd3-jshpn\" (UID: \"533b61f4-794e-48b8-aa70-4dd498211e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn"
Oct 30 00:52:16.400260 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.400241    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533b61f4-794e-48b8-aa70-4dd498211e12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adee9bd3-jshpn\" (UID: \"533b61f4-794e-48b8-aa70-4dd498211e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn"
Oct 30 00:52:16.501515 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.501468    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533b61f4-794e-48b8-aa70-4dd498211e12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adee9bd3-jshpn\" (UID: \"533b61f4-794e-48b8-aa70-4dd498211e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn"
Oct 30 00:52:16.501701 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.501559    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6fpf\" (UniqueName: \"kubernetes.io/projected/533b61f4-794e-48b8-aa70-4dd498211e12-kube-api-access-g6fpf\") pod \"maas-api-revoke-sub-adee9bd3-jshpn\" (UID: \"533b61f4-794e-48b8-aa70-4dd498211e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn"
Oct 30 00:52:16.501824 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.501773    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533b61f4-794e-48b8-aa70-4dd498211e12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adee9bd3-jshpn\" (UID: \"533b61f4-794e-48b8-aa70-4dd498211e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn"
Oct 30 00:52:16.502473 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.502430    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533b61f4-794e-48b8-aa70-4dd498211e12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adee9bd3-jshpn\" (UID: \"533b61f4-794e-48b8-aa70-4dd498211e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn"
Oct 30 00:52:16.504574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.504554    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533b61f4-794e-48b8-aa70-4dd498211e12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adee9bd3-jshpn\" (UID: \"533b61f4-794e-48b8-aa70-4dd498211e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn"
Oct 30 00:52:16.529854 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.529817    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6fpf\" (UniqueName: \"kubernetes.io/projected/533b61f4-794e-48b8-aa70-4dd498211e12-kube-api-access-g6fpf\") pod \"maas-api-revoke-sub-adee9bd3-jshpn\" (UID: \"533b61f4-794e-48b8-aa70-4dd498211e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn"
Oct 30 00:52:16.620032 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.619945    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn"
Oct 30 00:52:16.639475 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:52:16.639419    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:52:16.689914 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.689702    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6" event={"ID":"423014b8-4236-4135-8410-b53876b9a755","Type":"ContainerStarted","Data":"51c10930304963a999b30223beea8814b57165c2bd35961095528d6d30d63979"}
Oct 30 00:52:16.689914 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.689804    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:16.758045 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.757394    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6" podStartSLOduration=7.308372565 podStartE2EDuration="10.757374022s" totalImagesPullingTime="3.449001457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:16.748943329 +0000 UTC m=+1030.605851098" watchObservedRunningTime="2026-10-30 00:52:16.757374022 +0000 UTC m=+1030.614281778"
Oct 30 00:52:16.988024 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:52:16.987901    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533b61f4_794e_48b8_aa70_4dd498211e12.slice/crio-4d52562a5035a0890aa89ebe3d1737e9df41ad31dee26ec69f67ae69a91216b9 WatchSource:0}: Error finding container 4d52562a5035a0890aa89ebe3d1737e9df41ad31dee26ec69f67ae69a91216b9: Status 404 returned error can't find the container with id 4d52562a5035a0890aa89ebe3d1737e9df41ad31dee26ec69f67ae69a91216b9
Oct 30 00:52:16.988024 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:16.988022    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn"]
Oct 30 00:52:17.703988 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:17.703939    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn" event={"ID":"533b61f4-794e-48b8-aa70-4dd498211e12","Type":"ContainerStarted","Data":"44be71875a54fdc4e89300e5ca0b69e30cc31a22e7044c0ca046c6858bf80bf7"}
Oct 30 00:52:17.704184 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:17.704002    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn" event={"ID":"533b61f4-794e-48b8-aa70-4dd498211e12","Type":"ContainerStarted","Data":"4d52562a5035a0890aa89ebe3d1737e9df41ad31dee26ec69f67ae69a91216b9"}
Oct 30 00:52:17.706938 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:17.706916    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 00:52:17.749479 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:17.749421    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn" podStartSLOduration=1.749403467 podStartE2EDuration="1.749403467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:17.743697675 +0000 UTC m=+1031.600605443" watchObservedRunningTime="2026-10-30 00:52:17.749403467 +0000 UTC m=+1031.606311221"
Oct 30 00:52:18.714494 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:18.714456    6099 generic.go:356] "Generic (PLEG): container finished" podID="533b61f4-794e-48b8-aa70-4dd498211e12" containerID="44be71875a54fdc4e89300e5ca0b69e30cc31a22e7044c0ca046c6858bf80bf7" exitCode=0
Oct 30 00:52:18.714949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:18.714493    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn" event={"ID":"533b61f4-794e-48b8-aa70-4dd498211e12","Type":"ContainerDied","Data":"44be71875a54fdc4e89300e5ca0b69e30cc31a22e7044c0ca046c6858bf80bf7"}
Oct 30 00:52:19.904571 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:19.904545    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn"
Oct 30 00:52:19.972023 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:19.971990    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533b61f4-794e-48b8-aa70-4dd498211e12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533b61f4-794e-48b8-aa70-4dd498211e12-maas-api-openshift-ca\") pod \"533b61f4-794e-48b8-aa70-4dd498211e12\" (UID: \"533b61f4-794e-48b8-aa70-4dd498211e12\") "
Oct 30 00:52:19.972210 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:19.972185    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/533b61f4-794e-48b8-aa70-4dd498211e12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533b61f4-794e-48b8-aa70-4dd498211e12-maas-api-xks-ca\") pod \"533b61f4-794e-48b8-aa70-4dd498211e12\" (UID: \"533b61f4-794e-48b8-aa70-4dd498211e12\") "
Oct 30 00:52:19.972290 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:19.972270    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533b61f4-794e-48b8-aa70-4dd498211e12-kube-api-access-g6fpf\" (UniqueName: \"kubernetes.io/projected/533b61f4-794e-48b8-aa70-4dd498211e12-kube-api-access-g6fpf\") pod \"533b61f4-794e-48b8-aa70-4dd498211e12\" (UID: \"533b61f4-794e-48b8-aa70-4dd498211e12\") "
Oct 30 00:52:19.972434 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:19.972413    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533b61f4-794e-48b8-aa70-4dd498211e12-maas-api-openshift-ca" pod "533b61f4-794e-48b8-aa70-4dd498211e12" (UID: "533b61f4-794e-48b8-aa70-4dd498211e12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:52:19.973058 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:19.973033    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533b61f4-794e-48b8-aa70-4dd498211e12-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:19.974612 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:19.974556    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533b61f4-794e-48b8-aa70-4dd498211e12-maas-api-xks-ca" pod "533b61f4-794e-48b8-aa70-4dd498211e12" (UID: "533b61f4-794e-48b8-aa70-4dd498211e12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:52:19.975022 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:19.974983    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533b61f4-794e-48b8-aa70-4dd498211e12-kube-api-access-g6fpf" pod "533b61f4-794e-48b8-aa70-4dd498211e12" (UID: "533b61f4-794e-48b8-aa70-4dd498211e12"). InnerVolumeSpecName "kube-api-access-g6fpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:52:20.075184 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:20.075145    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533b61f4-794e-48b8-aa70-4dd498211e12-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:20.075184 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:20.075181    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6fpf\" (UniqueName: \"kubernetes.io/projected/533b61f4-794e-48b8-aa70-4dd498211e12-kube-api-access-g6fpf\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:20.733522 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:20.733487    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn"
Oct 30 00:52:20.733747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:20.733527    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn" event={"ID":"533b61f4-794e-48b8-aa70-4dd498211e12","Type":"ContainerDied","Data":"4d52562a5035a0890aa89ebe3d1737e9df41ad31dee26ec69f67ae69a91216b9"}
Oct 30 00:52:20.733747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:20.733572    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d52562a5035a0890aa89ebe3d1737e9df41ad31dee26ec69f67ae69a91216b9"
Oct 30 00:52:25.182487 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.182448    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt"]
Oct 30 00:52:25.183400 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.183360    6099 cpu_manager.go:414] "containerMap: removing container" podUID="533b61f4-794e-48b8-aa70-4dd498211e12" containerName="revoke-keys"
Oct 30 00:52:25.183400 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.183392    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533b61f4-794e-48b8-aa70-4dd498211e12" containerName="revoke-keys"
Oct 30 00:52:25.183739 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.183716    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="533b61f4-794e-48b8-aa70-4dd498211e12" containerName="revoke-keys"
Oct 30 00:52:25.189292 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.189269    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt"
Oct 30 00:52:25.213202 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.213156    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:52:25.213376 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.213241    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:52:25.213998 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.213919    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:52:25.218063 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.218025    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz9xn\" (UniqueName: \"kubernetes.io/projected/a438c487-876a-4b8b-87c2-2f13517e89eb-kube-api-access-bz9xn\") pod \"maas-api-revoke-sub-7a55081f-m8bbt\" (UID: \"a438c487-876a-4b8b-87c2-2f13517e89eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt"
Oct 30 00:52:25.218230 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.218147    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a438c487-876a-4b8b-87c2-2f13517e89eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a55081f-m8bbt\" (UID: \"a438c487-876a-4b8b-87c2-2f13517e89eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt"
Oct 30 00:52:25.218473 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.218446    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a438c487-876a-4b8b-87c2-2f13517e89eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a55081f-m8bbt\" (UID: \"a438c487-876a-4b8b-87c2-2f13517e89eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt"
Oct 30 00:52:25.223929 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.223901    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt"]
Oct 30 00:52:25.235722 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.234914    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:52:25.319927 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.319888    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a438c487-876a-4b8b-87c2-2f13517e89eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a55081f-m8bbt\" (UID: \"a438c487-876a-4b8b-87c2-2f13517e89eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt"
Oct 30 00:52:25.320132 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.320035    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9xn\" (UniqueName: \"kubernetes.io/projected/a438c487-876a-4b8b-87c2-2f13517e89eb-kube-api-access-bz9xn\") pod \"maas-api-revoke-sub-7a55081f-m8bbt\" (UID: \"a438c487-876a-4b8b-87c2-2f13517e89eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt"
Oct 30 00:52:25.320576 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.320548    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a438c487-876a-4b8b-87c2-2f13517e89eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a55081f-m8bbt\" (UID: \"a438c487-876a-4b8b-87c2-2f13517e89eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt"
Oct 30 00:52:25.322289 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.322262    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a438c487-876a-4b8b-87c2-2f13517e89eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a55081f-m8bbt\" (UID: \"a438c487-876a-4b8b-87c2-2f13517e89eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt"
Oct 30 00:52:25.322607 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.322585    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a438c487-876a-4b8b-87c2-2f13517e89eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a55081f-m8bbt\" (UID: \"a438c487-876a-4b8b-87c2-2f13517e89eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt"
Oct 30 00:52:25.343671 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.343646    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz9xn\" (UniqueName: \"kubernetes.io/projected/a438c487-876a-4b8b-87c2-2f13517e89eb-kube-api-access-bz9xn\") pod \"maas-api-revoke-sub-7a55081f-m8bbt\" (UID: \"a438c487-876a-4b8b-87c2-2f13517e89eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt"
Oct 30 00:52:25.539440 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.539413    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt"
Oct 30 00:52:25.871635 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:52:25.871475    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda438c487_876a_4b8b_87c2_2f13517e89eb.slice/crio-de86ef04144e07895fad7b58ff1fbb9660e58b2e5590007eed740497e5f3213b WatchSource:0}: Error finding container de86ef04144e07895fad7b58ff1fbb9660e58b2e5590007eed740497e5f3213b: Status 404 returned error can't find the container with id de86ef04144e07895fad7b58ff1fbb9660e58b2e5590007eed740497e5f3213b
Oct 30 00:52:25.872621 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:25.872545    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt"]
Oct 30 00:52:26.801074 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:26.801040    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt" event={"ID":"a438c487-876a-4b8b-87c2-2f13517e89eb","Type":"ContainerStarted","Data":"8a5da39bad16026be0bbd8703a25f5bbadde54305a817b1af57e8735a8e84834"}
Oct 30 00:52:26.801567 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:26.801101    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt" event={"ID":"a438c487-876a-4b8b-87c2-2f13517e89eb","Type":"ContainerStarted","Data":"de86ef04144e07895fad7b58ff1fbb9660e58b2e5590007eed740497e5f3213b"}
Oct 30 00:52:26.849746 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:26.848838    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt" podStartSLOduration=1.848820366 podStartE2EDuration="1.848820366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:26.842762294 +0000 UTC m=+1040.699670051" watchObservedRunningTime="2026-10-30 00:52:26.848820366 +0000 UTC m=+1040.705728121"
Oct 30 00:52:27.022729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:27.019425    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:52:27.811787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:27.811751    6099 generic.go:356] "Generic (PLEG): container finished" podID="a438c487-876a-4b8b-87c2-2f13517e89eb" containerID="8a5da39bad16026be0bbd8703a25f5bbadde54305a817b1af57e8735a8e84834" exitCode=0
Oct 30 00:52:27.812180 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:27.811818    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt" event={"ID":"a438c487-876a-4b8b-87c2-2f13517e89eb","Type":"ContainerDied","Data":"8a5da39bad16026be0bbd8703a25f5bbadde54305a817b1af57e8735a8e84834"}
Oct 30 00:52:29.807753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:29.805821    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:29.938748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:29.927939    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:29.938748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:29.935979    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:52:29.938748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:29.936191    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": context deadline exceeded"
Oct 30 00:52:30.253749 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.239943    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:30.481732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.470290    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:30.481732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.476097    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 00:52:30.544959 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.544923    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt"
Oct 30 00:52:30.600861 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.600818    6099 patch_prober.go:29] interesting pod/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:52:30.611884 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.611840    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh" podUID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:30.627738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.604972    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:30.870791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.829097    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a438c487-876a-4b8b-87c2-2f13517e89eb-kube-api-access-bz9xn\" (UniqueName: \"kubernetes.io/projected/a438c487-876a-4b8b-87c2-2f13517e89eb-kube-api-access-bz9xn\") pod \"a438c487-876a-4b8b-87c2-2f13517e89eb\" (UID: \"a438c487-876a-4b8b-87c2-2f13517e89eb\") "
Oct 30 00:52:31.020802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.948855    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a438c487-876a-4b8b-87c2-2f13517e89eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a438c487-876a-4b8b-87c2-2f13517e89eb-maas-api-xks-ca\") pod \"a438c487-876a-4b8b-87c2-2f13517e89eb\" (UID: \"a438c487-876a-4b8b-87c2-2f13517e89eb\") "
Oct 30 00:52:31.020802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.948934    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a438c487-876a-4b8b-87c2-2f13517e89eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a438c487-876a-4b8b-87c2-2f13517e89eb-maas-api-openshift-ca\") pod \"a438c487-876a-4b8b-87c2-2f13517e89eb\" (UID: \"a438c487-876a-4b8b-87c2-2f13517e89eb\") "
Oct 30 00:52:31.020802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.947859    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:52:31.020802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.928352    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:31.020802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.967788    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 00:52:31.020802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.968209    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:52:31.020802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.947817    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:31.020802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.988951    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a438c487-876a-4b8b-87c2-2f13517e89eb-maas-api-openshift-ca" pod "a438c487-876a-4b8b-87c2-2f13517e89eb" (UID: "a438c487-876a-4b8b-87c2-2f13517e89eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:52:31.020802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.947883    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:52:31.020802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.990114    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:52:31.020802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.990175    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:52:31.020802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:30.948434    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:31.129785 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.093252    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a438c487-876a-4b8b-87c2-2f13517e89eb-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:31.243736 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.243167    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:31.243736 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.243547    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:52:31.313848 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.313788    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:31.332145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.332100    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:31.391349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.360182    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a438c487-876a-4b8b-87c2-2f13517e89eb-kube-api-access-bz9xn" pod "a438c487-876a-4b8b-87c2-2f13517e89eb" (UID: "a438c487-876a-4b8b-87c2-2f13517e89eb"). InnerVolumeSpecName "kube-api-access-bz9xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:52:31.459741 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.446853    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a438c487-876a-4b8b-87c2-2f13517e89eb-maas-api-xks-ca" pod "a438c487-876a-4b8b-87c2-2f13517e89eb" (UID: "a438c487-876a-4b8b-87c2-2f13517e89eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:52:31.474253 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.474213    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 00:52:31.474869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.474844    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:52:31.525954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.525900    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:31.526623 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.526600    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:52:31.562417 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.562374    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a438c487-876a-4b8b-87c2-2f13517e89eb-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:31.564540 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.564514    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz9xn\" (UniqueName: \"kubernetes.io/projected/a438c487-876a-4b8b-87c2-2f13517e89eb-kube-api-access-bz9xn\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:31.640848 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.640799    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:31.655472 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.641470    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:52:31.698729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.695584    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt" event={"ID":"a438c487-876a-4b8b-87c2-2f13517e89eb","Type":"ContainerDied","Data":"de86ef04144e07895fad7b58ff1fbb9660e58b2e5590007eed740497e5f3213b"}
Oct 30 00:52:31.714823 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.714785    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de86ef04144e07895fad7b58ff1fbb9660e58b2e5590007eed740497e5f3213b"
Oct 30 00:52:31.737843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.695539    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt"
Oct 30 00:52:31.977922 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.977779    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:31.979671 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:31.979633    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:32.010980 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:32.010922    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": context deadline exceeded"
Oct 30 00:52:32.050732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:32.019552    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:52:32.081734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:32.062794    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:32.081734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:32.071893    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:32.154438 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:32.071409    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:32.535884 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:32.092024    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:52:32.838771 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:32.092546    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 00:52:32.840535 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:32.184006    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:52:32.844727 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:32.840824    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f" podUID="4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:32.873740 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:32.326054    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:32.874416 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:32.420885    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:52:32.887807 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:32.505134    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 00:52:32.914182 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:32.909864    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:32.920895 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:32.581475    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 00:52:33.530734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:33.507947    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 00:52:33.547735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:33.536287    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:52:33.593622 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:33.593574    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": context deadline exceeded"
Oct 30 00:52:33.594213 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:33.594171    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:33.733871 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:33.733746    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:52:33.739744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:33.739677    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 00:52:33.766184 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:33.763509    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:33.870129 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:33.866879    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 00:52:33.887728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:33.870450    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:52:33.887728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:33.870509    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt" podUID="6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:33.926069 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:33.926032    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:52:33.926653 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:33.926616    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:52:33.927132 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:33.927110    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 00:52:33.965593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:33.925974    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:52:34.056723 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.053579    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 00:52:34.076682 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:52:34.076651    6099 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.33s"
Oct 30 00:52:34.077061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.077040    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:52:34.077235 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.077222    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:52:34.090136 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.077691    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:52:34.090136 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.077828    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:52:34.138080 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.137988    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:52:34.177451 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.176397    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:52:34.290761 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.283546    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:52:34.318585 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.318557    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:52:34.330071 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.330046    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:52:34.497461 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.497380    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:52:34.512437 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.512408    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:52:34.552175 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.545495    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:52:34.552175 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.545569    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:52:34.578834 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.578794    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds"]
Oct 30 00:52:34.580268 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.580228    6099 cpu_manager.go:414] "containerMap: removing container" podUID="a438c487-876a-4b8b-87c2-2f13517e89eb" containerName="revoke-keys"
Oct 30 00:52:34.580268 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.580258    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a438c487-876a-4b8b-87c2-2f13517e89eb" containerName="revoke-keys"
Oct 30 00:52:34.580484 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.580443    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a438c487-876a-4b8b-87c2-2f13517e89eb" containerName="revoke-keys"
Oct 30 00:52:34.586329 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.586302    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds"
Oct 30 00:52:34.597324 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.596817    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds"]
Oct 30 00:52:34.602204 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.602183    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c4e6c558\""
Oct 30 00:52:34.602445 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.602430    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c4e6c558-dockercfg-khbbz\""
Oct 30 00:52:34.606950 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.604202    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t"]
Oct 30 00:52:34.612798 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.612776    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t"
Oct 30 00:52:34.620568 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.620548    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t"]
Oct 30 00:52:34.642844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.642809    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxcfr\" (UniqueName: \"kubernetes.io/projected/3f5d566c-1883-4a76-a994-8686faa32eee-kube-api-access-qxcfr\") pod \"payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t\" (UID: \"3f5d566c-1883-4a76-a994-8686faa32eee\") " pod="openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t"
Oct 30 00:52:34.643183 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.643153    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjqx6\" (UniqueName: \"kubernetes.io/projected/a794233b-4a04-49e2-8dc4-28b167e8cc95-kube-api-access-sjqx6\") pod \"payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds\" (UID: \"a794233b-4a04-49e2-8dc4-28b167e8cc95\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds"
Oct 30 00:52:34.643320 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.643251    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5d566c-1883-4a76-a994-8686faa32eee-plugins-config-volume\") pod \"payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t\" (UID: \"3f5d566c-1883-4a76-a994-8686faa32eee\") " pod="openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t"
Oct 30 00:52:34.643622 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.643593    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a794233b-4a04-49e2-8dc4-28b167e8cc95-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds\" (UID: \"a794233b-4a04-49e2-8dc4-28b167e8cc95\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds"
Oct 30 00:52:34.745268 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.745227    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxcfr\" (UniqueName: \"kubernetes.io/projected/3f5d566c-1883-4a76-a994-8686faa32eee-kube-api-access-qxcfr\") pod \"payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t\" (UID: \"3f5d566c-1883-4a76-a994-8686faa32eee\") " pod="openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t"
Oct 30 00:52:34.745523 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.745492    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjqx6\" (UniqueName: \"kubernetes.io/projected/a794233b-4a04-49e2-8dc4-28b167e8cc95-kube-api-access-sjqx6\") pod \"payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds\" (UID: \"a794233b-4a04-49e2-8dc4-28b167e8cc95\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds"
Oct 30 00:52:34.745655 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.745582    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5d566c-1883-4a76-a994-8686faa32eee-plugins-config-volume\") pod \"payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t\" (UID: \"3f5d566c-1883-4a76-a994-8686faa32eee\") " pod="openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t"
Oct 30 00:52:34.751920 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.746217    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a794233b-4a04-49e2-8dc4-28b167e8cc95-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds\" (UID: \"a794233b-4a04-49e2-8dc4-28b167e8cc95\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds"
Oct 30 00:52:34.751920 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.747151    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a794233b-4a04-49e2-8dc4-28b167e8cc95-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds\" (UID: \"a794233b-4a04-49e2-8dc4-28b167e8cc95\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds"
Oct 30 00:52:34.751920 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.747602    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5d566c-1883-4a76-a994-8686faa32eee-plugins-config-volume\") pod \"payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t\" (UID: \"3f5d566c-1883-4a76-a994-8686faa32eee\") " pod="openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t"
Oct 30 00:52:34.769860 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.769831    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxcfr\" (UniqueName: \"kubernetes.io/projected/3f5d566c-1883-4a76-a994-8686faa32eee-kube-api-access-qxcfr\") pod \"payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t\" (UID: \"3f5d566c-1883-4a76-a994-8686faa32eee\") " pod="openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t"
Oct 30 00:52:34.770048 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.770023    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjqx6\" (UniqueName: \"kubernetes.io/projected/a794233b-4a04-49e2-8dc4-28b167e8cc95-kube-api-access-sjqx6\") pod \"payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds\" (UID: \"a794233b-4a04-49e2-8dc4-28b167e8cc95\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds"
Oct 30 00:52:34.915667 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.915637    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds"
Oct 30 00:52:34.930364 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:34.930337    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t"
Oct 30 00:52:35.260490 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:52:35.260455    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda794233b_4a04_49e2_8dc4_28b167e8cc95.slice/crio-78917c44bcdb22d8a6292f752bd4fb1a23e1cbb4a12214c9040da4fbca180b98 WatchSource:0}: Error finding container 78917c44bcdb22d8a6292f752bd4fb1a23e1cbb4a12214c9040da4fbca180b98: Status 404 returned error can't find the container with id 78917c44bcdb22d8a6292f752bd4fb1a23e1cbb4a12214c9040da4fbca180b98
Oct 30 00:52:35.272309 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.272275    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds"]
Oct 30 00:52:35.484468 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:52:35.484434    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5d566c_1883_4a76_a994_8686faa32eee.slice/crio-b9fdb1530c5788fefe43727673fd25ef091c3fe6db72addefb6361f257bfb074 WatchSource:0}: Error finding container b9fdb1530c5788fefe43727673fd25ef091c3fe6db72addefb6361f257bfb074: Status 404 returned error can't find the container with id b9fdb1530c5788fefe43727673fd25ef091c3fe6db72addefb6361f257bfb074
Oct 30 00:52:35.486928 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.486905    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t"]
Oct 30 00:52:35.574193 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.574157    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9"]
Oct 30 00:52:35.578938 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.578917    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9"
Oct 30 00:52:35.593258 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.593235    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:52:35.593536 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.593293    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:52:35.593536 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.593376    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:52:35.593651 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.593552    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:52:35.598696 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.598673    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9"]
Oct 30 00:52:35.689010 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.688971    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mj7r\" (UniqueName: \"kubernetes.io/projected/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-kube-api-access-7mj7r\") pod \"maas-api-revoke-sub-aac1ae37-vrhv9\" (UID: \"94e6aab1-a6fd-4b49-b92d-7a7ab0982699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9"
Oct 30 00:52:35.689251 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.689230    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aac1ae37-vrhv9\" (UID: \"94e6aab1-a6fd-4b49-b92d-7a7ab0982699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9"
Oct 30 00:52:35.689335 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.689286    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aac1ae37-vrhv9\" (UID: \"94e6aab1-a6fd-4b49-b92d-7a7ab0982699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9"
Oct 30 00:52:35.722402 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.722374    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t" event={"ID":"3f5d566c-1883-4a76-a994-8686faa32eee","Type":"ContainerStarted","Data":"457d24b1bc5560bdbdebb7785cb4e314138dbc32c0a78fa82961fafa11884ef9"}
Oct 30 00:52:35.722581 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.722422    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t" event={"ID":"3f5d566c-1883-4a76-a994-8686faa32eee","Type":"ContainerStarted","Data":"b9fdb1530c5788fefe43727673fd25ef091c3fe6db72addefb6361f257bfb074"}
Oct 30 00:52:35.722581 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.722462    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t"
Oct 30 00:52:35.724099 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.724078    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds" event={"ID":"a794233b-4a04-49e2-8dc4-28b167e8cc95","Type":"ContainerStarted","Data":"43e07cd47c06e7d41fda9919dcf56fc9368b2cfc33ef2e57fe6f984fa74bc715"}
Oct 30 00:52:35.724195 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.724110    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds" event={"ID":"a794233b-4a04-49e2-8dc4-28b167e8cc95","Type":"ContainerStarted","Data":"78917c44bcdb22d8a6292f752bd4fb1a23e1cbb4a12214c9040da4fbca180b98"}
Oct 30 00:52:35.724254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.724195    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds"
Oct 30 00:52:35.763826 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.763750    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t" podStartSLOduration=1.7637397350000001 podStartE2EDuration="1.763739735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:35.763025402 +0000 UTC m=+1049.619933157" watchObservedRunningTime="2026-10-30 00:52:35.763739735 +0000 UTC m=+1049.620647489"
Oct 30 00:52:35.790755 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.790693    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mj7r\" (UniqueName: \"kubernetes.io/projected/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-kube-api-access-7mj7r\") pod \"maas-api-revoke-sub-aac1ae37-vrhv9\" (UID: \"94e6aab1-a6fd-4b49-b92d-7a7ab0982699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9"
Oct 30 00:52:35.790909 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.790896    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aac1ae37-vrhv9\" (UID: \"94e6aab1-a6fd-4b49-b92d-7a7ab0982699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9"
Oct 30 00:52:35.790980 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.790958    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aac1ae37-vrhv9\" (UID: \"94e6aab1-a6fd-4b49-b92d-7a7ab0982699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9"
Oct 30 00:52:35.791562 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.791542    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aac1ae37-vrhv9\" (UID: \"94e6aab1-a6fd-4b49-b92d-7a7ab0982699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9"
Oct 30 00:52:35.793273 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.793254    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aac1ae37-vrhv9\" (UID: \"94e6aab1-a6fd-4b49-b92d-7a7ab0982699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9"
Oct 30 00:52:35.807289 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.807237    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds" podStartSLOduration=1.807220981 podStartE2EDuration="1.807220981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:35.801559573 +0000 UTC m=+1049.658467329" watchObservedRunningTime="2026-10-30 00:52:35.807220981 +0000 UTC m=+1049.664128737"
Oct 30 00:52:35.815919 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.815893    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mj7r\" (UniqueName: \"kubernetes.io/projected/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-kube-api-access-7mj7r\") pod \"maas-api-revoke-sub-aac1ae37-vrhv9\" (UID: \"94e6aab1-a6fd-4b49-b92d-7a7ab0982699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9"
Oct 30 00:52:35.892505 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:35.892483    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9"
Oct 30 00:52:36.209332 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:52:36.209299    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e6aab1_a6fd_4b49_b92d_7a7ab0982699.slice/crio-9b23f7a4e1925a7a64d70c7b614580131d7d8ba209b29068366bb6e16aafb9e1 WatchSource:0}: Error finding container 9b23f7a4e1925a7a64d70c7b614580131d7d8ba209b29068366bb6e16aafb9e1: Status 404 returned error can't find the container with id 9b23f7a4e1925a7a64d70c7b614580131d7d8ba209b29068366bb6e16aafb9e1
Oct 30 00:52:36.211309 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:36.211282    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9"]
Oct 30 00:52:36.735412 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:36.735296    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9" event={"ID":"94e6aab1-a6fd-4b49-b92d-7a7ab0982699","Type":"ContainerStarted","Data":"38c473367134ee20adf0969ac32ed654a1c2c0530025cc5545a696fcbcad1696"}
Oct 30 00:52:36.735412 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:36.735377    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9" event={"ID":"94e6aab1-a6fd-4b49-b92d-7a7ab0982699","Type":"ContainerStarted","Data":"9b23f7a4e1925a7a64d70c7b614580131d7d8ba209b29068366bb6e16aafb9e1"}
Oct 30 00:52:36.776635 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:36.776584    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9" podStartSLOduration=1.77656573 podStartE2EDuration="1.77656573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:36.773995435 +0000 UTC m=+1050.630903190" watchObservedRunningTime="2026-10-30 00:52:36.77656573 +0000 UTC m=+1050.633473486"
Oct 30 00:52:37.746901 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:37.746855    6099 generic.go:356] "Generic (PLEG): container finished" podID="94e6aab1-a6fd-4b49-b92d-7a7ab0982699" containerID="38c473367134ee20adf0969ac32ed654a1c2c0530025cc5545a696fcbcad1696" exitCode=0
Oct 30 00:52:37.747410 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:37.746913    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9" event={"ID":"94e6aab1-a6fd-4b49-b92d-7a7ab0982699","Type":"ContainerDied","Data":"38c473367134ee20adf0969ac32ed654a1c2c0530025cc5545a696fcbcad1696"}
Oct 30 00:52:38.932593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:38.932568    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9"
Oct 30 00:52:39.039800 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:39.039771    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-kube-api-access-7mj7r\" (UniqueName: \"kubernetes.io/projected/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-kube-api-access-7mj7r\") pod \"94e6aab1-a6fd-4b49-b92d-7a7ab0982699\" (UID: \"94e6aab1-a6fd-4b49-b92d-7a7ab0982699\") "
Oct 30 00:52:39.039947 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:39.039848    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-maas-api-openshift-ca\") pod \"94e6aab1-a6fd-4b49-b92d-7a7ab0982699\" (UID: \"94e6aab1-a6fd-4b49-b92d-7a7ab0982699\") "
Oct 30 00:52:39.039947 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:39.039893    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-maas-api-xks-ca\") pod \"94e6aab1-a6fd-4b49-b92d-7a7ab0982699\" (UID: \"94e6aab1-a6fd-4b49-b92d-7a7ab0982699\") "
Oct 30 00:52:39.040229 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:39.040198    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-maas-api-openshift-ca" pod "94e6aab1-a6fd-4b49-b92d-7a7ab0982699" (UID: "94e6aab1-a6fd-4b49-b92d-7a7ab0982699"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:52:39.040671 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:39.040653    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:39.042019 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:39.041999    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-kube-api-access-7mj7r" pod "94e6aab1-a6fd-4b49-b92d-7a7ab0982699" (UID: "94e6aab1-a6fd-4b49-b92d-7a7ab0982699"). InnerVolumeSpecName "kube-api-access-7mj7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:52:39.042019 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:39.042007    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-maas-api-xks-ca" pod "94e6aab1-a6fd-4b49-b92d-7a7ab0982699" (UID: "94e6aab1-a6fd-4b49-b92d-7a7ab0982699"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:52:39.142147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:39.142123    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mj7r\" (UniqueName: \"kubernetes.io/projected/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-kube-api-access-7mj7r\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:39.142147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:39.142149    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e6aab1-a6fd-4b49-b92d-7a7ab0982699-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:39.769904 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:39.769860    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9" event={"ID":"94e6aab1-a6fd-4b49-b92d-7a7ab0982699","Type":"ContainerDied","Data":"9b23f7a4e1925a7a64d70c7b614580131d7d8ba209b29068366bb6e16aafb9e1"}
Oct 30 00:52:39.770103 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:39.769931    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b23f7a4e1925a7a64d70c7b614580131d7d8ba209b29068366bb6e16aafb9e1"
Oct 30 00:52:39.770103 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:39.770001    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9"
Oct 30 00:52:42.838820 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:42.838784    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7"]
Oct 30 00:52:42.839651 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:42.839626    6099 cpu_manager.go:414] "containerMap: removing container" podUID="94e6aab1-a6fd-4b49-b92d-7a7ab0982699" containerName="revoke-keys"
Oct 30 00:52:42.839651 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:42.839646    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e6aab1-a6fd-4b49-b92d-7a7ab0982699" containerName="revoke-keys"
Oct 30 00:52:42.839936 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:42.839913    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e6aab1-a6fd-4b49-b92d-7a7ab0982699" containerName="revoke-keys"
Oct 30 00:52:42.844472 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:42.844454    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7"
Oct 30 00:52:42.864314 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:42.864288    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7"]
Oct 30 00:52:42.869515 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:42.869485    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:52:42.869515 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:42.869496    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:52:42.869515 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:42.869505    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:52:42.869735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:42.869532    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:52:42.999972 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:42.999935    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24vdn\" (UniqueName: \"kubernetes.io/projected/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-kube-api-access-24vdn\") pod \"maas-api-revoke-sub-49070d8b-4znv7\" (UID: \"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7"
Oct 30 00:52:43.000137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:43.000106    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49070d8b-4znv7\" (UID: \"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7"
Oct 30 00:52:43.000189 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:43.000146    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49070d8b-4znv7\" (UID: \"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7"
Oct 30 00:52:43.102050 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:43.101965    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24vdn\" (UniqueName: \"kubernetes.io/projected/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-kube-api-access-24vdn\") pod \"maas-api-revoke-sub-49070d8b-4znv7\" (UID: \"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7"
Oct 30 00:52:43.102242 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:43.102082    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49070d8b-4znv7\" (UID: \"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7"
Oct 30 00:52:43.102242 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:43.102107    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49070d8b-4znv7\" (UID: \"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7"
Oct 30 00:52:43.102751 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:43.102730    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49070d8b-4znv7\" (UID: \"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7"
Oct 30 00:52:43.104444 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:43.104425    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49070d8b-4znv7\" (UID: \"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7"
Oct 30 00:52:43.124840 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:43.124813    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24vdn\" (UniqueName: \"kubernetes.io/projected/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-kube-api-access-24vdn\") pod \"maas-api-revoke-sub-49070d8b-4znv7\" (UID: \"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7"
Oct 30 00:52:43.168283 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:43.168255    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7"
Oct 30 00:52:43.478343 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:52:43.478316    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd1bd97_d7ff_44dd_88fd_bc4cfae19210.slice/crio-47e9f0e95ea62b8358e9320ecdd4d3b3d04ed3d5cbb9ebe6090ea4bf5781113f WatchSource:0}: Error finding container 47e9f0e95ea62b8358e9320ecdd4d3b3d04ed3d5cbb9ebe6090ea4bf5781113f: Status 404 returned error can't find the container with id 47e9f0e95ea62b8358e9320ecdd4d3b3d04ed3d5cbb9ebe6090ea4bf5781113f
Oct 30 00:52:43.478986 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:43.478959    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7"]
Oct 30 00:52:43.808964 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:43.808932    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7" event={"ID":"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210","Type":"ContainerStarted","Data":"ffee9dde426f61747e891c4b89ec7fe751d9bfa22e98f183735fe38f2a081203"}
Oct 30 00:52:43.809160 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:43.808987    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7" event={"ID":"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210","Type":"ContainerStarted","Data":"47e9f0e95ea62b8358e9320ecdd4d3b3d04ed3d5cbb9ebe6090ea4bf5781113f"}
Oct 30 00:52:43.854413 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:43.854366    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7" podStartSLOduration=1.854352964 podStartE2EDuration="1.854352964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:43.848627416 +0000 UTC m=+1057.705535171" watchObservedRunningTime="2026-10-30 00:52:43.854352964 +0000 UTC m=+1057.711260736"
Oct 30 00:52:44.821182 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:44.821142    6099 generic.go:356] "Generic (PLEG): container finished" podID="0fd1bd97-d7ff-44dd-88fd-bc4cfae19210" containerID="ffee9dde426f61747e891c4b89ec7fe751d9bfa22e98f183735fe38f2a081203" exitCode=0
Oct 30 00:52:44.821352 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:44.821228    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7" event={"ID":"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210","Type":"ContainerDied","Data":"ffee9dde426f61747e891c4b89ec7fe751d9bfa22e98f183735fe38f2a081203"}
Oct 30 00:52:45.988220 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:45.988195    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7"
Oct 30 00:52:46.150973 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:46.150882    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-kube-api-access-24vdn\" (UniqueName: \"kubernetes.io/projected/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-kube-api-access-24vdn\") pod \"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210\" (UID: \"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210\") "
Oct 30 00:52:46.151147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:46.151030    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-maas-api-openshift-ca\") pod \"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210\" (UID: \"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210\") "
Oct 30 00:52:46.151147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:46.151096    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-maas-api-xks-ca\") pod \"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210\" (UID: \"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210\") "
Oct 30 00:52:46.151373 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:46.151348    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-maas-api-openshift-ca" pod "0fd1bd97-d7ff-44dd-88fd-bc4cfae19210" (UID: "0fd1bd97-d7ff-44dd-88fd-bc4cfae19210"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:52:46.151563 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:46.151532    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:46.153052 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:46.153029    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-maas-api-xks-ca" pod "0fd1bd97-d7ff-44dd-88fd-bc4cfae19210" (UID: "0fd1bd97-d7ff-44dd-88fd-bc4cfae19210"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:52:46.153156 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:46.153136    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-kube-api-access-24vdn" pod "0fd1bd97-d7ff-44dd-88fd-bc4cfae19210" (UID: "0fd1bd97-d7ff-44dd-88fd-bc4cfae19210"). InnerVolumeSpecName "kube-api-access-24vdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:52:46.253536 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:46.253502    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:46.253536 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:46.253536    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24vdn\" (UniqueName: \"kubernetes.io/projected/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210-kube-api-access-24vdn\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:46.630776 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:52:46.630737    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:52:46.738646 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:46.738603    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds"
Oct 30 00:52:46.743669 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:46.743642    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t"
Oct 30 00:52:46.893944 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:46.893855    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7"
Oct 30 00:52:46.894210 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:46.893854    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7" event={"ID":"0fd1bd97-d7ff-44dd-88fd-bc4cfae19210","Type":"ContainerDied","Data":"47e9f0e95ea62b8358e9320ecdd4d3b3d04ed3d5cbb9ebe6090ea4bf5781113f"}
Oct 30 00:52:46.894343 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:46.894322    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47e9f0e95ea62b8358e9320ecdd4d3b3d04ed3d5cbb9ebe6090ea4bf5781113f"
Oct 30 00:52:47.966052 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:47.966011    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp"]
Oct 30 00:52:47.967393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:47.967365    6099 cpu_manager.go:414] "containerMap: removing container" podUID="0fd1bd97-d7ff-44dd-88fd-bc4cfae19210" containerName="revoke-keys"
Oct 30 00:52:47.967393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:47.967392    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd1bd97-d7ff-44dd-88fd-bc4cfae19210" containerName="revoke-keys"
Oct 30 00:52:47.967653 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:47.967638    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd1bd97-d7ff-44dd-88fd-bc4cfae19210" containerName="revoke-keys"
Oct 30 00:52:47.973289 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:47.973261    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp"
Oct 30 00:52:47.990270 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:47.990242    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp"]
Oct 30 00:52:47.999887 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:47.999002    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:52:47.999887 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:47.999160    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:52:47.999887 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:47.999194    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:52:47.999887 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:47.999841    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:52:48.092051 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.092013    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a853723b-1bf6-4458-ab94-056b85e63a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74bb6183-lxxfp\" (UID: \"a853723b-1bf6-4458-ab94-056b85e63a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp"
Oct 30 00:52:48.092248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.092137    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29f5z\" (UniqueName: \"kubernetes.io/projected/a853723b-1bf6-4458-ab94-056b85e63a31-kube-api-access-29f5z\") pod \"maas-api-revoke-sub-74bb6183-lxxfp\" (UID: \"a853723b-1bf6-4458-ab94-056b85e63a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp"
Oct 30 00:52:48.092248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.092172    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a853723b-1bf6-4458-ab94-056b85e63a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74bb6183-lxxfp\" (UID: \"a853723b-1bf6-4458-ab94-056b85e63a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp"
Oct 30 00:52:48.194116 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.194079    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a853723b-1bf6-4458-ab94-056b85e63a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74bb6183-lxxfp\" (UID: \"a853723b-1bf6-4458-ab94-056b85e63a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp"
Oct 30 00:52:48.194288 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.194151    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29f5z\" (UniqueName: \"kubernetes.io/projected/a853723b-1bf6-4458-ab94-056b85e63a31-kube-api-access-29f5z\") pod \"maas-api-revoke-sub-74bb6183-lxxfp\" (UID: \"a853723b-1bf6-4458-ab94-056b85e63a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp"
Oct 30 00:52:48.194288 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.194178    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a853723b-1bf6-4458-ab94-056b85e63a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74bb6183-lxxfp\" (UID: \"a853723b-1bf6-4458-ab94-056b85e63a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp"
Oct 30 00:52:48.194852 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.194826    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a853723b-1bf6-4458-ab94-056b85e63a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74bb6183-lxxfp\" (UID: \"a853723b-1bf6-4458-ab94-056b85e63a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp"
Oct 30 00:52:48.196574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.196547    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a853723b-1bf6-4458-ab94-056b85e63a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74bb6183-lxxfp\" (UID: \"a853723b-1bf6-4458-ab94-056b85e63a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp"
Oct 30 00:52:48.217163 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.217085    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29f5z\" (UniqueName: \"kubernetes.io/projected/a853723b-1bf6-4458-ab94-056b85e63a31-kube-api-access-29f5z\") pod \"maas-api-revoke-sub-74bb6183-lxxfp\" (UID: \"a853723b-1bf6-4458-ab94-056b85e63a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp"
Oct 30 00:52:48.291016 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.290985    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp"
Oct 30 00:52:48.594809 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:52:48.594772    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda853723b_1bf6_4458_ab94_056b85e63a31.slice/crio-b725f9b76f2ca1d1e2693626d97b9d4592faff17cc1f1f770238fb75c5c02dc7 WatchSource:0}: Error finding container b725f9b76f2ca1d1e2693626d97b9d4592faff17cc1f1f770238fb75c5c02dc7: Status 404 returned error can't find the container with id b725f9b76f2ca1d1e2693626d97b9d4592faff17cc1f1f770238fb75c5c02dc7
Oct 30 00:52:48.598647 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.598621    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp"]
Oct 30 00:52:48.781207 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.781180    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"]
Oct 30 00:52:48.787444 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.787423    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:48.819761 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.815928    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"]
Oct 30 00:52:48.830058 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.830023    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"]
Oct 30 00:52:48.838159 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.838128    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:48.869459 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.869431    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"]
Oct 30 00:52:48.906467 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.906423    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:48.906650 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.906495    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:48.906650 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.906557    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:48.906854 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.906687    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:48.906854 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.906757    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbs6q\" (UniqueName: \"kubernetes.io/projected/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-kube-api-access-wbs6q\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:48.906972 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.906859    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:48.906972 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.906889    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:48.906972 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.906950    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:48.907138 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.906998    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:48.958411 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:48.958378    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp" event={"ID":"a853723b-1bf6-4458-ab94-056b85e63a31","Type":"ContainerStarted","Data":"b725f9b76f2ca1d1e2693626d97b9d4592faff17cc1f1f770238fb75c5c02dc7"}
Oct 30 00:52:49.008785 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.008749    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.009241 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.008813    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.009241 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.008840    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4srvh\" (UniqueName: \"kubernetes.io/projected/4be8330a-cd36-47fa-b660-05a5796f3473-kube-api-access-4srvh\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.009241 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.008874    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.009241 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009009    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.009241 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009085    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.009241 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009140    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-data\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.009241 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009190    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.009241 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009211    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be8330a-cd36-47fa-b660-05a5796f3473-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.009647 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009286    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.009647 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009335    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.009647 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009351    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.009647 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009379    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.009647 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009387    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-workload-certs\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.009647 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009396    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.009647 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009577    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-credential-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.009647 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009587    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.009647 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009608    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istiod-ca-cert\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.009647 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009647    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be8330a-cd36-47fa-b660-05a5796f3473-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.010044 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009726    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-workload-socket\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.010044 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009768    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.010044 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009786    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbs6q\" (UniqueName: \"kubernetes.io/projected/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-kube-api-access-wbs6q\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.010044 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.009857    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be8330a-cd36-47fa-b660-05a5796f3473-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.012008 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.011981    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-envoy\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.012232 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.012215    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-podinfo\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.050090 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.050046    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbs6q\" (UniqueName: \"kubernetes.io/projected/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-kube-api-access-wbs6q\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.051652 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.051605    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-token\") pod \"e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") " pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.086351 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.086321    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr"]
Oct 30 00:52:49.091517 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.091490    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03da01dc-kkmhr"]
Oct 30 00:52:49.102270 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.102245    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:49.111322 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.111222    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.111322 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.111284    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.111477 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.111380    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.111477 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.111423    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be8330a-cd36-47fa-b660-05a5796f3473-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.111568 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.111516    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be8330a-cd36-47fa-b660-05a5796f3473-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.111610 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.111582    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.111648 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.111609    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4srvh\" (UniqueName: \"kubernetes.io/projected/4be8330a-cd36-47fa-b660-05a5796f3473-kube-api-access-4srvh\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.111803 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.111784    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.111873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.111818    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be8330a-cd36-47fa-b660-05a5796f3473-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.112838 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.112815    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-istio-data\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.113118 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.113071    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-workload-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.113171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.113109    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be8330a-cd36-47fa-b660-05a5796f3473-istiod-ca-cert\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.113319 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.113292    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-credential-socket\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.114750 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.114688    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-workload-certs\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.115587 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.115559    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-istio-envoy\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.115953 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.115933    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be8330a-cd36-47fa-b660-05a5796f3473-istio-podinfo\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.148295 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.148260    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be8330a-cd36-47fa-b660-05a5796f3473-istio-token\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.148468 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.148442    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4srvh\" (UniqueName: \"kubernetes.io/projected/4be8330a-cd36-47fa-b660-05a5796f3473-kube-api-access-4srvh\") pod \"e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") " pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.178249 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.178218    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:49.440133 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.437497    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"]
Oct 30 00:52:49.440133 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:52:49.437857    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f0f0a1_18c0_4b97_b24e_ce5289a0cce8.slice/crio-b989c27ad2c8ca45bb3874888aae6f3aa007cdf2eed0ae42123560f50510f1a5 WatchSource:0}: Error finding container b989c27ad2c8ca45bb3874888aae6f3aa007cdf2eed0ae42123560f50510f1a5: Status 404 returned error can't find the container with id b989c27ad2c8ca45bb3874888aae6f3aa007cdf2eed0ae42123560f50510f1a5
Oct 30 00:52:49.441804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.440976    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:52:49.441804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.441060    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:52:49.441804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.441113    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:52:49.508416 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:52:49.508384    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be8330a_cd36_47fa_b660_05a5796f3473.slice/crio-56384f77eeea2c4e436564f510bf4878eb3186f4077b037d7587b3ccee09062f WatchSource:0}: Error finding container 56384f77eeea2c4e436564f510bf4878eb3186f4077b037d7587b3ccee09062f: Status 404 returned error can't find the container with id 56384f77eeea2c4e436564f510bf4878eb3186f4077b037d7587b3ccee09062f
Oct 30 00:52:49.511955 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.511932    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"]
Oct 30 00:52:49.512048 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.511973    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:52:49.512089 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.512059    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:52:49.512126 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.512109    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:52:49.975982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.974061    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b" event={"ID":"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8","Type":"ContainerStarted","Data":"334158f39fa6186d7e4dbced2fde62fee9e9be8a792168c1912ac1f92a6e2272"}
Oct 30 00:52:49.975982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.974127    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b" event={"ID":"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8","Type":"ContainerStarted","Data":"b989c27ad2c8ca45bb3874888aae6f3aa007cdf2eed0ae42123560f50510f1a5"}
Oct 30 00:52:49.977596 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.977564    6099 generic.go:356] "Generic (PLEG): container finished" podID="a853723b-1bf6-4458-ab94-056b85e63a31" containerID="7506e3000c8c947cd74aec5505a4fa4163a469e6d9a589307e69a52ff618b356" exitCode=0
Oct 30 00:52:49.979451 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.977870    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp" event={"ID":"a853723b-1bf6-4458-ab94-056b85e63a31","Type":"ContainerDied","Data":"7506e3000c8c947cd74aec5505a4fa4163a469e6d9a589307e69a52ff618b356"}
Oct 30 00:52:49.982019 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.981960    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z" event={"ID":"4be8330a-cd36-47fa-b660-05a5796f3473","Type":"ContainerStarted","Data":"1c6f5888e46c6bc2fca989973604ab88689290376538279d8732173f48f3a58b"}
Oct 30 00:52:49.982163 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:49.982024    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z" event={"ID":"4be8330a-cd36-47fa-b660-05a5796f3473","Type":"ContainerStarted","Data":"56384f77eeea2c4e436564f510bf4878eb3186f4077b037d7587b3ccee09062f"}
Oct 30 00:52:50.026888 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:50.026839    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b" podStartSLOduration=2.026825936 podStartE2EDuration="2.026825936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:50.021925149 +0000 UTC m=+1063.878832904" watchObservedRunningTime="2026-10-30 00:52:50.026825936 +0000 UTC m=+1063.883733690"
Oct 30 00:52:50.103738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:50.102998    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:50.105468 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:50.105136    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 00:52:50.105468 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:50.105204    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b" podUID="a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 00:52:50.133218 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:50.133161    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z" podStartSLOduration=2.13314723 podStartE2EDuration="2.13314723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:50.077140641 +0000 UTC m=+1063.934048397" watchObservedRunningTime="2026-10-30 00:52:50.13314723 +0000 UTC m=+1063.990054985"
Oct 30 00:52:50.180738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:50.180337    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:50.203734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:50.184827    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 00:52:50.203734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:50.184890    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z" podUID="4be8330a-cd36-47fa-b660-05a5796f3473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 00:52:50.263553 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:50.263475    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0" path="/var/lib/kubelet/pods/fb7f8ba0-bbd4-4a2b-9cdc-fb763111f4b0/volumes"
Oct 30 00:52:51.123357 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:51.122905    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 00:52:51.123357 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:51.122975    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b" podUID="a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 00:52:51.230454 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:51.228829    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 00:52:51.230454 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:51.228890    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z" podUID="4be8330a-cd36-47fa-b660-05a5796f3473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 00:52:51.385637 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:51.385604    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp"
Oct 30 00:52:51.527201 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:51.527159    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a853723b-1bf6-4458-ab94-056b85e63a31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a853723b-1bf6-4458-ab94-056b85e63a31-maas-api-openshift-ca\") pod \"a853723b-1bf6-4458-ab94-056b85e63a31\" (UID: \"a853723b-1bf6-4458-ab94-056b85e63a31\") "
Oct 30 00:52:51.527405 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:51.527233    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a853723b-1bf6-4458-ab94-056b85e63a31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a853723b-1bf6-4458-ab94-056b85e63a31-maas-api-xks-ca\") pod \"a853723b-1bf6-4458-ab94-056b85e63a31\" (UID: \"a853723b-1bf6-4458-ab94-056b85e63a31\") "
Oct 30 00:52:51.527405 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:51.527266    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a853723b-1bf6-4458-ab94-056b85e63a31-kube-api-access-29f5z\" (UniqueName: \"kubernetes.io/projected/a853723b-1bf6-4458-ab94-056b85e63a31-kube-api-access-29f5z\") pod \"a853723b-1bf6-4458-ab94-056b85e63a31\" (UID: \"a853723b-1bf6-4458-ab94-056b85e63a31\") "
Oct 30 00:52:51.528639 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:51.528605    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a853723b-1bf6-4458-ab94-056b85e63a31-maas-api-openshift-ca" pod "a853723b-1bf6-4458-ab94-056b85e63a31" (UID: "a853723b-1bf6-4458-ab94-056b85e63a31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:52:51.530857 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:51.530830    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a853723b-1bf6-4458-ab94-056b85e63a31-kube-api-access-29f5z" pod "a853723b-1bf6-4458-ab94-056b85e63a31" (UID: "a853723b-1bf6-4458-ab94-056b85e63a31"). InnerVolumeSpecName "kube-api-access-29f5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:52:51.532878 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:51.532852    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a853723b-1bf6-4458-ab94-056b85e63a31-maas-api-xks-ca" pod "a853723b-1bf6-4458-ab94-056b85e63a31" (UID: "a853723b-1bf6-4458-ab94-056b85e63a31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:52:51.630631 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:51.630594    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a853723b-1bf6-4458-ab94-056b85e63a31-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:51.630631 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:51.630624    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a853723b-1bf6-4458-ab94-056b85e63a31-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:51.630631 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:51.630634    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29f5z\" (UniqueName: \"kubernetes.io/projected/a853723b-1bf6-4458-ab94-056b85e63a31-kube-api-access-29f5z\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:52:52.115359 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:52.115321    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 00:52:52.115676 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:52.115630    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b" podUID="a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 00:52:52.163750 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:52.155207    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp"
Oct 30 00:52:52.194740 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:52.194523    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 00:52:52.194740 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:52.194580    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z" podUID="4be8330a-cd36-47fa-b660-05a5796f3473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 00:52:52.227733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:52.226935    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp" event={"ID":"a853723b-1bf6-4458-ab94-056b85e63a31","Type":"ContainerDied","Data":"b725f9b76f2ca1d1e2693626d97b9d4592faff17cc1f1f770238fb75c5c02dc7"}
Oct 30 00:52:52.227733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:52.227008    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b725f9b76f2ca1d1e2693626d97b9d4592faff17cc1f1f770238fb75c5c02dc7"
Oct 30 00:52:53.117771 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:53.112638    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 00:52:53.117771 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:53.112720    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b" podUID="a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 00:52:53.184835 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:53.184800    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 00:52:53.185461 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:53.185430    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z" podUID="4be8330a-cd36-47fa-b660-05a5796f3473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 00:52:54.114731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:54.113932    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 00:52:54.114731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:54.114015    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b" podUID="a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 00:52:54.189729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:54.187287    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 00:52:54.189729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:54.187357    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z" podUID="4be8330a-cd36-47fa-b660-05a5796f3473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 00:52:55.109731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:55.104105    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 00:52:55.109731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:55.104170    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b" podUID="a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 00:52:55.193290 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:55.193252    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 00:52:55.193907 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:55.193871    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z" podUID="4be8330a-cd36-47fa-b660-05a5796f3473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 00:52:56.039101 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:56.039061    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c"]
Oct 30 00:52:56.046149 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:56.046123    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0726217b-8nd6c"]
Oct 30 00:52:56.102806 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:56.102770    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 00:52:56.102965 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:56.102836    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b" podUID="a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 00:52:56.180420 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:56.180384    6099 patch_prober.go:29] interesting pod/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 00:52:56.180599 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:56.180441    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z" podUID="4be8330a-cd36-47fa-b660-05a5796f3473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 00:52:56.368242 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:56.368151    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ee9d15a-a18e-44b7-8b16-2e30fd8ce423" path="/var/lib/kubelet/pods/1ee9d15a-a18e-44b7-8b16-2e30fd8ce423/volumes"
Oct 30 00:52:57.289833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:57.289794    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:57.290394 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:57.290371    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:57.291262 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:57.291241    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:52:57.461908 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:57.461874    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:57.462357 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:57.462293    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:52:57.463153 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:52:57.463127    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 00:53:03.089731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:03.079760    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc"]
Oct 30 00:53:03.113769 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:03.113730    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf7b88e8-9v8bc"]
Oct 30 00:53:03.606731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:03.605820    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:03.934915 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:03.889917    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:53:03.952734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:03.915782    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:04.247784 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:04.069436    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:53:04.342064 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:04.334968    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:04.425017 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:04.419940    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:04.463434 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:04.463306    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="597d5e40-34d9-4362-acb6-447fa9664d80" path="/var/lib/kubelet/pods/597d5e40-34d9-4362-acb6-447fa9664d80/volumes"
Oct 30 00:53:04.682097 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:04.682051    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 00:53:04.831874 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:04.682344    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:04.851943 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:04.699837    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 00:53:04.877731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:04.852425    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:53:04.997817 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:04.991832    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:04.997817 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:04.993065    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:53:05.085732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:05.067937    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"]
Oct 30 00:53:05.085732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:05.069142    6099 cpu_manager.go:414] "containerMap: removing container" podUID="a853723b-1bf6-4458-ab94-056b85e63a31" containerName="revoke-keys"
Oct 30 00:53:05.085732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:05.069165    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a853723b-1bf6-4458-ab94-056b85e63a31" containerName="revoke-keys"
Oct 30 00:53:05.085732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:05.069355    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a853723b-1bf6-4458-ab94-056b85e63a31" containerName="revoke-keys"
Oct 30 00:53:05.214730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:05.196810    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:05.242770 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:05.225907    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": context deadline exceeded"
Oct 30 00:53:05.254212 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:05.254114    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:05.284088 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:05.284044    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:05.301735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:05.283830    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:05.401822 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:05.357812    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:05.431783 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:05.412481    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 00:53:05.466733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:05.460789    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:53:05.898934 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:05.887892    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:05.919808 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:05.892692    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": context deadline exceeded"
Oct 30 00:53:06.315731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:06.311236    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:53:06.459137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:06.459087    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:06.550963 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:05.978865    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:06.580787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:05.951602    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:06.591030 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:06.479289    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:06.890933 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:05.951295    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:53:06.891102 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:06.891019    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:53:06.891176 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:06.891119    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:53:06.891243 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:06.891204    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:53:06.891243 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:06.891231    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"]
Oct 30 00:53:06.948128 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:06.891362    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:53:06.948128 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:06.891645    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:53:06.948128 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:06.891752    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:53:06.948128 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:06.939321    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:06.948128 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:06.939672    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:06.981739 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:06.973817    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 00:53:07.105775 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.096925    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:53:07.105775 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.097051    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 00:53:07.109858 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.109804    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:53:07.455360 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.455313    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:53:07.456044 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.456012    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded"
Oct 30 00:53:07.579731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.573828    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 00:53:07.579731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.578779    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:07.628775 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.628691    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:07.722786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.708120    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:53:07.752105 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.752061    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49bd9\" (UniqueName: \"kubernetes.io/projected/e8270463-2deb-44c8-baa7-9ccfa2167945-kube-api-access-49bd9\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:07.752394 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.752358    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:07.752635 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.752615    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:07.756831 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.756785    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:07.757064 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.757045    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:07.757457 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.757435    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8270463-2deb-44c8-baa7-9ccfa2167945-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:07.757687 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.757667    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:07.761858 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.761835    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:07.762152 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.762110    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:07.956735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.946346    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:07.957319 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.957273    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:07.997594 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:07.997506    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:08.321785 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.315601    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49bd9\" (UniqueName: \"kubernetes.io/projected/e8270463-2deb-44c8-baa7-9ccfa2167945-kube-api-access-49bd9\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:08.413734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.137422    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:53:08.413734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.141631    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:08.413734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.079502    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-data\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:08.413734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.141873    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:08.413734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.141686    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv" podUID="5c8179b0-96f4-4647-b926-5969b4afecd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Oct 30 00:53:08.413734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.201150    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:08.413734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.299794    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:08.413734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.390466    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:08.413734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.390525    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:08.413734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.390564    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:08.413734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.390675    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:08.413734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.393843    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8270463-2deb-44c8-baa7-9ccfa2167945-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:08.413734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.393995    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:08.413734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.394178    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:08.558729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.532696    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8270463-2deb-44c8-baa7-9ccfa2167945-istiod-ca-cert\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:08.700788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.679443    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-workload-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:08.700788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.695216    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-credential-socket\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:08.700788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.698416    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-token\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:08.768856 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.768825    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:53:08.805283 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.805247    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-envoy\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:08.808094 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.808063    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49bd9\" (UniqueName: \"kubernetes.io/projected/e8270463-2deb-44c8-baa7-9ccfa2167945-kube-api-access-49bd9\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:08.816869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:08.816844    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-workload-certs\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:09.181910 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.181865    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 00:53:09.182537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.182506    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/health\": context deadline exceeded"
Oct 30 00:53:09.519448 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.519351    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 00:53:09.532514 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.532474    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:09.536734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.533234    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": context deadline exceeded"
Oct 30 00:53:09.561739 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.539008    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 00:53:09.562590 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.549295    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:53:09.563027 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.549329    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 00:53:09.581261 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.581219    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:53:09.581903 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.581868    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 00:53:09.643867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.619389    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:09.644862 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.644824    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded"
Oct 30 00:53:09.645060 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.645038    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:53:09.746382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.746351    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:53:09.856722 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.856671    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:53:09.895076 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.895049    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:53:09.900441 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.900419    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:53:09.909536 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.909510    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:53:09.965403 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.959784    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"]
Oct 30 00:53:09.987847 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:09.987810    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded"
Oct 30 00:53:10.002995 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.000853    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-podinfo\") pod \"e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") " pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:10.050696 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.050659    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.051545 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.050621    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:53:10.064872 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.062509    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"]
Oct 30 00:53:10.067885 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.067850    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:53:10.069948 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.066508    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-ea2547-openshift-default-dockercfg-g9jmp\""
Oct 30 00:53:10.073078 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.073032    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:53:10.076953 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.076928    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:53:10.077061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.077005    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:53:10.077061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.077059    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:53:10.198500 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.198411    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:10.209894 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.209856    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-workload-certs\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.210066 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.210046    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5pst\" (UniqueName: \"kubernetes.io/projected/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-kube-api-access-t5pst\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.210198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.210182    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-credential-socket\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.210282 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.210262    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-token\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.211820 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.211786    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istiod-ca-cert\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.211947 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.211893    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-podinfo\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.212029 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.211964    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-data\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.212029 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.212013    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-workload-socket\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.214788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.213034    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-envoy\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.314825 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.314788    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-token\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.314943 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.314925    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istiod-ca-cert\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.315021 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.315005    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-podinfo\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.315089 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.315074    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-data\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.315133 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.315117    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-workload-socket\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.315220 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.315203    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-envoy\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.315308 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.315293    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-workload-certs\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.315443 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.315427    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5pst\" (UniqueName: \"kubernetes.io/projected/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-kube-api-access-t5pst\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.315510 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.315496    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-credential-socket\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.319659 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.317066    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-credential-socket\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.319659 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.318110    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-data\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.319659 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.318416    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-workload-socket\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.319659 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.319050    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istiod-ca-cert\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.319659 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.319425    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-workload-certs\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.324016 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.323973    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-envoy\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.339771 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.339294    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-podinfo\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.344292 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.342361    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-token\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.344292 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.344171    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5pst\" (UniqueName: \"kubernetes.io/projected/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-kube-api-access-t5pst\") pod \"e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") " pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.396852 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.396821    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:10.580372 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:53:10.580337    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8270463_2deb_44c8_baa7_9ccfa2167945.slice/crio-c51c0c06714027817fdba5479b0a38e328ec2d733ddcaa0aedd557ce14feba36 WatchSource:0}: Error finding container c51c0c06714027817fdba5479b0a38e328ec2d733ddcaa0aedd557ce14feba36: Status 404 returned error can't find the container with id c51c0c06714027817fdba5479b0a38e328ec2d733ddcaa0aedd557ce14feba36
Oct 30 00:53:10.582202 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.581580    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"]
Oct 30 00:53:10.583676 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.583638    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:53:10.583826 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.583784    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:53:10.583899 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.583837    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:53:10.712094 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:53:10.712021    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1acc2b4d_a946_42e3_aacd_4519ad6bd9e6.slice/crio-fb68f9e3394dde4dc185a6013388a31e422dc76c271a6d1c117aed8ac6add4f0 WatchSource:0}: Error finding container fb68f9e3394dde4dc185a6013388a31e422dc76c271a6d1c117aed8ac6add4f0: Status 404 returned error can't find the container with id fb68f9e3394dde4dc185a6013388a31e422dc76c271a6d1c117aed8ac6add4f0
Oct 30 00:53:10.714034 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.714014    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"]
Oct 30 00:53:10.715315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.715279    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:53:10.715393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.715378    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:53:10.715443 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:10.715427    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:53:11.032003 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:11.031901    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx" event={"ID":"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6","Type":"ContainerStarted","Data":"2aab72d2a83ca606bff17e89d4f1ce26a87b69cd92b97c409d5e9eed830d959c"}
Oct 30 00:53:11.032003 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:11.031977    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx" event={"ID":"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6","Type":"ContainerStarted","Data":"fb68f9e3394dde4dc185a6013388a31e422dc76c271a6d1c117aed8ac6add4f0"}
Oct 30 00:53:11.034082 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:11.034053    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9" event={"ID":"e8270463-2deb-44c8-baa7-9ccfa2167945","Type":"ContainerStarted","Data":"0bef5f1ae66940b0f1fd5b45bc0f5da8d4b6b3fb1e2c7c78558e52f9e94262b6"}
Oct 30 00:53:11.034210 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:11.034101    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9" event={"ID":"e8270463-2deb-44c8-baa7-9ccfa2167945","Type":"ContainerStarted","Data":"c51c0c06714027817fdba5479b0a38e328ec2d733ddcaa0aedd557ce14feba36"}
Oct 30 00:53:11.092863 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:11.092792    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx" podStartSLOduration=3.092775039 podStartE2EDuration="3.092775039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:11.087439276 +0000 UTC m=+1084.944347029" watchObservedRunningTime="2026-10-30 00:53:11.092775039 +0000 UTC m=+1084.949682796"
Oct 30 00:53:11.137296 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:11.137249    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9" podStartSLOduration=7.137237087 podStartE2EDuration="7.137237087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:11.131922442 +0000 UTC m=+1084.988830190" watchObservedRunningTime="2026-10-30 00:53:11.137237087 +0000 UTC m=+1084.994144842"
Oct 30 00:53:11.198949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:11.198914    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:11.200896 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:11.200869    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 00:53:11.201031 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:11.200923    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9" podUID="e8270463-2deb-44c8-baa7-9ccfa2167945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 00:53:11.399593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:11.398062    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:11.410980 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:11.410941    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:12.032474 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:12.032434    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5"]
Oct 30 00:53:12.043418 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:12.043346    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:12.043592 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:12.043442    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb9196ee-4j7j5"]
Oct 30 00:53:12.044236 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:12.044214    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:53:12.199423 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:12.199385    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 00:53:12.199585 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:12.199454    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9" podUID="e8270463-2deb-44c8-baa7-9ccfa2167945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 00:53:12.263845 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:12.263812    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51921f0c-2f68-4220-ba0d-c95ae82ad03f" path="/var/lib/kubelet/pods/51921f0c-2f68-4220-ba0d-c95ae82ad03f/volumes"
Oct 30 00:53:13.033659 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:13.033624    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs"]
Oct 30 00:53:13.045218 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:13.045190    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef64c63-lltcs"]
Oct 30 00:53:13.199106 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:13.199059    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 00:53:13.199270 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:13.199136    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9" podUID="e8270463-2deb-44c8-baa7-9ccfa2167945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 00:53:14.199777 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:14.199743    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 00:53:14.200207 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:14.199804    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9" podUID="e8270463-2deb-44c8-baa7-9ccfa2167945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 00:53:14.263647 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:14.263605    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d1b67c-e24b-4b75-bf55-c2c7c7d57285" path="/var/lib/kubelet/pods/c5d1b67c-e24b-4b75-bf55-c2c7c7d57285/volumes"
Oct 30 00:53:15.199640 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:15.199598    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 00:53:15.199870 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:15.199663    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9" podUID="e8270463-2deb-44c8-baa7-9ccfa2167945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 00:53:16.199543 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.199504    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 00:53:16.199738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.199568    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9" podUID="e8270463-2deb-44c8-baa7-9ccfa2167945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 00:53:16.426288 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.426247    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d"]
Oct 30 00:53:16.432193 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.432174    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d"
Oct 30 00:53:16.451901 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.451842    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d"]
Oct 30 00:53:16.455062 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.455023    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:53:16.455166 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.455139    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:53:16.455467 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.455449    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:53:16.455603 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.455585    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:53:16.545588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.545548    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaae5a9d-72aa-4817-8135-8f6ba657e420-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d\" (UID: \"aaae5a9d-72aa-4817-8135-8f6ba657e420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d"
Oct 30 00:53:16.545920 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.545897    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf8d6\" (UniqueName: \"kubernetes.io/projected/aaae5a9d-72aa-4817-8135-8f6ba657e420-kube-api-access-vf8d6\") pod \"maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d\" (UID: \"aaae5a9d-72aa-4817-8135-8f6ba657e420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d"
Oct 30 00:53:16.546363 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.546343    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaae5a9d-72aa-4817-8135-8f6ba657e420-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d\" (UID: \"aaae5a9d-72aa-4817-8135-8f6ba657e420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d"
Oct 30 00:53:16.644206 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:53:16.644162    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:53:16.648745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.648470    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaae5a9d-72aa-4817-8135-8f6ba657e420-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d\" (UID: \"aaae5a9d-72aa-4817-8135-8f6ba657e420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d"
Oct 30 00:53:16.648745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.648567    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaae5a9d-72aa-4817-8135-8f6ba657e420-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d\" (UID: \"aaae5a9d-72aa-4817-8135-8f6ba657e420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d"
Oct 30 00:53:16.648745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.648619    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf8d6\" (UniqueName: \"kubernetes.io/projected/aaae5a9d-72aa-4817-8135-8f6ba657e420-kube-api-access-vf8d6\") pod \"maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d\" (UID: \"aaae5a9d-72aa-4817-8135-8f6ba657e420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d"
Oct 30 00:53:16.649130 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.649109    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaae5a9d-72aa-4817-8135-8f6ba657e420-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d\" (UID: \"aaae5a9d-72aa-4817-8135-8f6ba657e420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d"
Oct 30 00:53:16.654115 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.654087    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaae5a9d-72aa-4817-8135-8f6ba657e420-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d\" (UID: \"aaae5a9d-72aa-4817-8135-8f6ba657e420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d"
Oct 30 00:53:16.696774 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.695207    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf8d6\" (UniqueName: \"kubernetes.io/projected/aaae5a9d-72aa-4817-8135-8f6ba657e420-kube-api-access-vf8d6\") pod \"maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d\" (UID: \"aaae5a9d-72aa-4817-8135-8f6ba657e420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d"
Oct 30 00:53:16.750771 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:16.750685    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d"
Oct 30 00:53:17.311168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:17.286252    6099 patch_prober.go:29] interesting pod/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 00:53:17.311168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:17.286342    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9" podUID="e8270463-2deb-44c8-baa7-9ccfa2167945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 00:53:17.465026 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:17.464962    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d"]
Oct 30 00:53:18.482727 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:18.482014    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d" event={"ID":"aaae5a9d-72aa-4817-8135-8f6ba657e420","Type":"ContainerStarted","Data":"88322e852822ec19933bbe4224ac3deccfbcf63716205979d39d93811bea801e"}
Oct 30 00:53:18.482727 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:18.482088    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d" event={"ID":"aaae5a9d-72aa-4817-8135-8f6ba657e420","Type":"ContainerStarted","Data":"19c6518ccfc76db2f0d12789345bf2275c5976bc7967cff7a56f34cab09af79f"}
Oct 30 00:53:18.532073 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:18.531997    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d" podStartSLOduration=2.531978335 podStartE2EDuration="2.531978335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:18.522058841 +0000 UTC m=+1092.378966599" watchObservedRunningTime="2026-10-30 00:53:18.531978335 +0000 UTC m=+1092.388886091"
Oct 30 00:53:18.661339 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:18.658933    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:18.661339 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:18.660410    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:18.662105 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:18.662062    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:53:19.494114 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:19.494077    6099 generic.go:356] "Generic (PLEG): container finished" podID="aaae5a9d-72aa-4817-8135-8f6ba657e420" containerID="88322e852822ec19933bbe4224ac3deccfbcf63716205979d39d93811bea801e" exitCode=0
Oct 30 00:53:19.494598 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:19.494169    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d" event={"ID":"aaae5a9d-72aa-4817-8135-8f6ba657e420","Type":"ContainerDied","Data":"88322e852822ec19933bbe4224ac3deccfbcf63716205979d39d93811bea801e"}
Oct 30 00:53:20.669146 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:20.669120    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d"
Oct 30 00:53:20.745872 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:20.745841    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaae5a9d-72aa-4817-8135-8f6ba657e420-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaae5a9d-72aa-4817-8135-8f6ba657e420-maas-api-openshift-ca\") pod \"aaae5a9d-72aa-4817-8135-8f6ba657e420\" (UID: \"aaae5a9d-72aa-4817-8135-8f6ba657e420\") "
Oct 30 00:53:20.746052 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:20.746033    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaae5a9d-72aa-4817-8135-8f6ba657e420-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaae5a9d-72aa-4817-8135-8f6ba657e420-maas-api-xks-ca\") pod \"aaae5a9d-72aa-4817-8135-8f6ba657e420\" (UID: \"aaae5a9d-72aa-4817-8135-8f6ba657e420\") "
Oct 30 00:53:20.746231 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:20.746208    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaae5a9d-72aa-4817-8135-8f6ba657e420-maas-api-openshift-ca" pod "aaae5a9d-72aa-4817-8135-8f6ba657e420" (UID: "aaae5a9d-72aa-4817-8135-8f6ba657e420"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:20.746322 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:20.746285    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaae5a9d-72aa-4817-8135-8f6ba657e420-kube-api-access-vf8d6\" (UniqueName: \"kubernetes.io/projected/aaae5a9d-72aa-4817-8135-8f6ba657e420-kube-api-access-vf8d6\") pod \"aaae5a9d-72aa-4817-8135-8f6ba657e420\" (UID: \"aaae5a9d-72aa-4817-8135-8f6ba657e420\") "
Oct 30 00:53:20.747041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:20.747020    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaae5a9d-72aa-4817-8135-8f6ba657e420-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:53:20.748665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:20.748642    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaae5a9d-72aa-4817-8135-8f6ba657e420-maas-api-xks-ca" pod "aaae5a9d-72aa-4817-8135-8f6ba657e420" (UID: "aaae5a9d-72aa-4817-8135-8f6ba657e420"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:53:20.748928 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:20.748909    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaae5a9d-72aa-4817-8135-8f6ba657e420-kube-api-access-vf8d6" pod "aaae5a9d-72aa-4817-8135-8f6ba657e420" (UID: "aaae5a9d-72aa-4817-8135-8f6ba657e420"). InnerVolumeSpecName "kube-api-access-vf8d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:20.848812 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:20.848774    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf8d6\" (UniqueName: \"kubernetes.io/projected/aaae5a9d-72aa-4817-8135-8f6ba657e420-kube-api-access-vf8d6\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:53:20.848812 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:20.848811    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaae5a9d-72aa-4817-8135-8f6ba657e420-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:53:21.578730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:21.577688    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d" event={"ID":"aaae5a9d-72aa-4817-8135-8f6ba657e420","Type":"ContainerDied","Data":"19c6518ccfc76db2f0d12789345bf2275c5976bc7967cff7a56f34cab09af79f"}
Oct 30 00:53:21.578730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:21.577786    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19c6518ccfc76db2f0d12789345bf2275c5976bc7967cff7a56f34cab09af79f"
Oct 30 00:53:21.578730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:21.577860    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d"
Oct 30 00:53:22.812473 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:22.812441    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln"]
Oct 30 00:53:22.813226 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:22.813211    6099 cpu_manager.go:414] "containerMap: removing container" podUID="aaae5a9d-72aa-4817-8135-8f6ba657e420" containerName="revoke-keys"
Oct 30 00:53:22.813279 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:22.813227    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaae5a9d-72aa-4817-8135-8f6ba657e420" containerName="revoke-keys"
Oct 30 00:53:22.813334 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:22.813323    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaae5a9d-72aa-4817-8135-8f6ba657e420" containerName="revoke-keys"
Oct 30 00:53:22.817652 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:22.817631    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln"
Oct 30 00:53:22.844405 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:22.844378    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:53:22.844524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:22.844403    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:53:22.846388 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:22.846368    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:53:22.846499 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:22.846429    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:53:22.855386 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:22.855362    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln"]
Oct 30 00:53:23.001886 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:23.001853    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acb3744-7d81-4e2c-a358-3768bfcbac34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-74b07396-7ghln\" (UID: \"1acb3744-7d81-4e2c-a358-3768bfcbac34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln"
Oct 30 00:53:23.002117 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:23.002096    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acb3744-7d81-4e2c-a358-3768bfcbac34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-74b07396-7ghln\" (UID: \"1acb3744-7d81-4e2c-a358-3768bfcbac34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln"
Oct 30 00:53:23.002245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:23.002231    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h867x\" (UniqueName: \"kubernetes.io/projected/1acb3744-7d81-4e2c-a358-3768bfcbac34-kube-api-access-h867x\") pod \"maas-api-revoke-keys-e2e-derive-74b07396-7ghln\" (UID: \"1acb3744-7d81-4e2c-a358-3768bfcbac34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln"
Oct 30 00:53:23.033957 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:23.033926    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds"]
Oct 30 00:53:23.040055 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:23.040032    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27aaa04e-vlwds"]
Oct 30 00:53:23.103551 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:23.103471    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h867x\" (UniqueName: \"kubernetes.io/projected/1acb3744-7d81-4e2c-a358-3768bfcbac34-kube-api-access-h867x\") pod \"maas-api-revoke-keys-e2e-derive-74b07396-7ghln\" (UID: \"1acb3744-7d81-4e2c-a358-3768bfcbac34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln"
Oct 30 00:53:23.103672 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:23.103651    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acb3744-7d81-4e2c-a358-3768bfcbac34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-74b07396-7ghln\" (UID: \"1acb3744-7d81-4e2c-a358-3768bfcbac34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln"
Oct 30 00:53:23.103851 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:23.103834    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acb3744-7d81-4e2c-a358-3768bfcbac34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-74b07396-7ghln\" (UID: \"1acb3744-7d81-4e2c-a358-3768bfcbac34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln"
Oct 30 00:53:23.104421 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:23.104403    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acb3744-7d81-4e2c-a358-3768bfcbac34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-74b07396-7ghln\" (UID: \"1acb3744-7d81-4e2c-a358-3768bfcbac34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln"
Oct 30 00:53:23.106091 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:23.106070    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acb3744-7d81-4e2c-a358-3768bfcbac34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-74b07396-7ghln\" (UID: \"1acb3744-7d81-4e2c-a358-3768bfcbac34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln"
Oct 30 00:53:23.126496 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:23.126473    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h867x\" (UniqueName: \"kubernetes.io/projected/1acb3744-7d81-4e2c-a358-3768bfcbac34-kube-api-access-h867x\") pod \"maas-api-revoke-keys-e2e-derive-74b07396-7ghln\" (UID: \"1acb3744-7d81-4e2c-a358-3768bfcbac34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln"
Oct 30 00:53:23.137360 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:23.137329    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln"
Oct 30 00:53:23.452794 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:23.452770    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln"]
Oct 30 00:53:23.596463 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:23.596436    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln" event={"ID":"1acb3744-7d81-4e2c-a358-3768bfcbac34","Type":"ContainerStarted","Data":"6601144c91a680590997f86cdbe741e3e9f29ae0fa72b8d45dae37c52e858caf"}
Oct 30 00:53:24.266786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:24.266746    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a3f843a-ea97-4bb7-8a43-05a44b264942" path="/var/lib/kubelet/pods/5a3f843a-ea97-4bb7-8a43-05a44b264942/volumes"
Oct 30 00:53:24.608152 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:24.608113    6099 generic.go:356] "Generic (PLEG): container finished" podID="1acb3744-7d81-4e2c-a358-3768bfcbac34" containerID="0a4e00bba51101601851cdfd2d40c26b5bc765856313ba16e3d43f2009c6715b" exitCode=0
Oct 30 00:53:24.608292 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:24.608183    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln" event={"ID":"1acb3744-7d81-4e2c-a358-3768bfcbac34","Type":"ContainerDied","Data":"0a4e00bba51101601851cdfd2d40c26b5bc765856313ba16e3d43f2009c6715b"}
Oct 30 00:53:25.799987 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:25.799963    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln"
Oct 30 00:53:25.856382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:25.856350    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1acb3744-7d81-4e2c-a358-3768bfcbac34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acb3744-7d81-4e2c-a358-3768bfcbac34-maas-api-openshift-ca\") pod \"1acb3744-7d81-4e2c-a358-3768bfcbac34\" (UID: \"1acb3744-7d81-4e2c-a358-3768bfcbac34\") "
Oct 30 00:53:25.856559 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:25.856457    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1acb3744-7d81-4e2c-a358-3768bfcbac34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acb3744-7d81-4e2c-a358-3768bfcbac34-maas-api-xks-ca\") pod \"1acb3744-7d81-4e2c-a358-3768bfcbac34\" (UID: \"1acb3744-7d81-4e2c-a358-3768bfcbac34\") "
Oct 30 00:53:25.856636 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:25.856568    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1acb3744-7d81-4e2c-a358-3768bfcbac34-kube-api-access-h867x\" (UniqueName: \"kubernetes.io/projected/1acb3744-7d81-4e2c-a358-3768bfcbac34-kube-api-access-h867x\") pod \"1acb3744-7d81-4e2c-a358-3768bfcbac34\" (UID: \"1acb3744-7d81-4e2c-a358-3768bfcbac34\") "
Oct 30 00:53:25.856825 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:25.856793    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1acb3744-7d81-4e2c-a358-3768bfcbac34-maas-api-openshift-ca" pod "1acb3744-7d81-4e2c-a358-3768bfcbac34" (UID: "1acb3744-7d81-4e2c-a358-3768bfcbac34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:25.857542 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:25.857516    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acb3744-7d81-4e2c-a358-3768bfcbac34-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:53:25.859396 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:25.859351    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acb3744-7d81-4e2c-a358-3768bfcbac34-kube-api-access-h867x" pod "1acb3744-7d81-4e2c-a358-3768bfcbac34" (UID: "1acb3744-7d81-4e2c-a358-3768bfcbac34"). InnerVolumeSpecName "kube-api-access-h867x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:25.859524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:25.859433    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1acb3744-7d81-4e2c-a358-3768bfcbac34-maas-api-xks-ca" pod "1acb3744-7d81-4e2c-a358-3768bfcbac34" (UID: "1acb3744-7d81-4e2c-a358-3768bfcbac34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:53:25.959161 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:25.959127    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acb3744-7d81-4e2c-a358-3768bfcbac34-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:53:25.959161 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:25.959155    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h867x\" (UniqueName: \"kubernetes.io/projected/1acb3744-7d81-4e2c-a358-3768bfcbac34-kube-api-access-h867x\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:53:26.079671 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.079631    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j"]
Oct 30 00:53:26.080886 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.080855    6099 cpu_manager.go:414] "containerMap: removing container" podUID="1acb3744-7d81-4e2c-a358-3768bfcbac34" containerName="revoke-keys"
Oct 30 00:53:26.081102 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.081059    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1acb3744-7d81-4e2c-a358-3768bfcbac34" containerName="revoke-keys"
Oct 30 00:53:26.081390 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.081369    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1acb3744-7d81-4e2c-a358-3768bfcbac34" containerName="revoke-keys"
Oct 30 00:53:26.086473 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.086442    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j"
Oct 30 00:53:26.138861 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.138822    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j"]
Oct 30 00:53:26.163564 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.163521    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7e8dda6-863d-4264-966a-a922a45fa930-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j\" (UID: \"f7e8dda6-863d-4264-966a-a922a45fa930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j"
Oct 30 00:53:26.163752 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.163623    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m5sc\" (UniqueName: \"kubernetes.io/projected/f7e8dda6-863d-4264-966a-a922a45fa930-kube-api-access-6m5sc\") pod \"maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j\" (UID: \"f7e8dda6-863d-4264-966a-a922a45fa930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j"
Oct 30 00:53:26.164676 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.163994    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7e8dda6-863d-4264-966a-a922a45fa930-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j\" (UID: \"f7e8dda6-863d-4264-966a-a922a45fa930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j"
Oct 30 00:53:26.265591 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.265553    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m5sc\" (UniqueName: \"kubernetes.io/projected/f7e8dda6-863d-4264-966a-a922a45fa930-kube-api-access-6m5sc\") pod \"maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j\" (UID: \"f7e8dda6-863d-4264-966a-a922a45fa930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j"
Oct 30 00:53:26.265954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.265933    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7e8dda6-863d-4264-966a-a922a45fa930-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j\" (UID: \"f7e8dda6-863d-4264-966a-a922a45fa930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j"
Oct 30 00:53:26.266078 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.266060    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7e8dda6-863d-4264-966a-a922a45fa930-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j\" (UID: \"f7e8dda6-863d-4264-966a-a922a45fa930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j"
Oct 30 00:53:26.266770 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.266739    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7e8dda6-863d-4264-966a-a922a45fa930-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j\" (UID: \"f7e8dda6-863d-4264-966a-a922a45fa930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j"
Oct 30 00:53:26.269239 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.269210    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7e8dda6-863d-4264-966a-a922a45fa930-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j\" (UID: \"f7e8dda6-863d-4264-966a-a922a45fa930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j"
Oct 30 00:53:26.290484 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.290461    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m5sc\" (UniqueName: \"kubernetes.io/projected/f7e8dda6-863d-4264-966a-a922a45fa930-kube-api-access-6m5sc\") pod \"maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j\" (UID: \"f7e8dda6-863d-4264-966a-a922a45fa930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j"
Oct 30 00:53:26.451985 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.451916    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j"
Oct 30 00:53:26.626855 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.626828    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln"
Oct 30 00:53:26.626855 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.626833    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln" event={"ID":"1acb3744-7d81-4e2c-a358-3768bfcbac34","Type":"ContainerDied","Data":"6601144c91a680590997f86cdbe741e3e9f29ae0fa72b8d45dae37c52e858caf"}
Oct 30 00:53:26.627061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.626890    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6601144c91a680590997f86cdbe741e3e9f29ae0fa72b8d45dae37c52e858caf"
Oct 30 00:53:26.755839 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:53:26.755802    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e8dda6_863d_4264_966a_a922a45fa930.slice/crio-88f348fbc3dcfdeab9465a8553987e503437d934e946a02786d97552ebde5f24 WatchSource:0}: Error finding container 88f348fbc3dcfdeab9465a8553987e503437d934e946a02786d97552ebde5f24: Status 404 returned error can't find the container with id 88f348fbc3dcfdeab9465a8553987e503437d934e946a02786d97552ebde5f24
Oct 30 00:53:26.758056 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:26.758033    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j"]
Oct 30 00:53:27.638783 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:27.638744    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j" event={"ID":"f7e8dda6-863d-4264-966a-a922a45fa930","Type":"ContainerStarted","Data":"4879c730b4703774380426dc34cbea7614bcbd394c872deba21fbad00afbcd41"}
Oct 30 00:53:27.639198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:27.638819    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j" event={"ID":"f7e8dda6-863d-4264-966a-a922a45fa930","Type":"ContainerStarted","Data":"88f348fbc3dcfdeab9465a8553987e503437d934e946a02786d97552ebde5f24"}
Oct 30 00:53:27.678794 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:27.678743    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j" podStartSLOduration=1.678727225 podStartE2EDuration="1.678727225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:27.677814864 +0000 UTC m=+1101.534722619" watchObservedRunningTime="2026-10-30 00:53:27.678727225 +0000 UTC m=+1101.535634977"
Oct 30 00:53:28.033144 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:28.033057    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs"]
Oct 30 00:53:28.038725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:28.038682    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj"]
Oct 30 00:53:28.044633 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:28.044593    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffc215d6-dqqrs"]
Oct 30 00:53:28.048905 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:28.048884    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1286c8aa-6dgtj"]
Oct 30 00:53:28.268395 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:28.268357    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="767c1c10-93bd-4ac3-8b4c-4869626a7b89" path="/var/lib/kubelet/pods/767c1c10-93bd-4ac3-8b4c-4869626a7b89/volumes"
Oct 30 00:53:28.269120 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:28.269081    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811321cc-7afa-4b87-a18a-b930be1be4c4" path="/var/lib/kubelet/pods/811321cc-7afa-4b87-a18a-b930be1be4c4/volumes"
Oct 30 00:53:28.658773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:28.653864    6099 generic.go:356] "Generic (PLEG): container finished" podID="f7e8dda6-863d-4264-966a-a922a45fa930" containerID="4879c730b4703774380426dc34cbea7614bcbd394c872deba21fbad00afbcd41" exitCode=0
Oct 30 00:53:28.658773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:28.654200    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j" event={"ID":"f7e8dda6-863d-4264-966a-a922a45fa930","Type":"ContainerDied","Data":"4879c730b4703774380426dc34cbea7614bcbd394c872deba21fbad00afbcd41"}
Oct 30 00:53:29.876919 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:29.876891    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j"
Oct 30 00:53:29.921406 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:29.921327    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7e8dda6-863d-4264-966a-a922a45fa930-kube-api-access-6m5sc\" (UniqueName: \"kubernetes.io/projected/f7e8dda6-863d-4264-966a-a922a45fa930-kube-api-access-6m5sc\") pod \"f7e8dda6-863d-4264-966a-a922a45fa930\" (UID: \"f7e8dda6-863d-4264-966a-a922a45fa930\") "
Oct 30 00:53:29.921564 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:29.921433    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7e8dda6-863d-4264-966a-a922a45fa930-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7e8dda6-863d-4264-966a-a922a45fa930-maas-api-openshift-ca\") pod \"f7e8dda6-863d-4264-966a-a922a45fa930\" (UID: \"f7e8dda6-863d-4264-966a-a922a45fa930\") "
Oct 30 00:53:29.921564 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:29.921497    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7e8dda6-863d-4264-966a-a922a45fa930-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7e8dda6-863d-4264-966a-a922a45fa930-maas-api-xks-ca\") pod \"f7e8dda6-863d-4264-966a-a922a45fa930\" (UID: \"f7e8dda6-863d-4264-966a-a922a45fa930\") "
Oct 30 00:53:29.923177 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:29.923141    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e8dda6-863d-4264-966a-a922a45fa930-maas-api-openshift-ca" pod "f7e8dda6-863d-4264-966a-a922a45fa930" (UID: "f7e8dda6-863d-4264-966a-a922a45fa930"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:29.925231 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:29.925207    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7e8dda6-863d-4264-966a-a922a45fa930-maas-api-xks-ca" pod "f7e8dda6-863d-4264-966a-a922a45fa930" (UID: "f7e8dda6-863d-4264-966a-a922a45fa930"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:53:29.925992 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:29.925917    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e8dda6-863d-4264-966a-a922a45fa930-kube-api-access-6m5sc" pod "f7e8dda6-863d-4264-966a-a922a45fa930" (UID: "f7e8dda6-863d-4264-966a-a922a45fa930"). InnerVolumeSpecName "kube-api-access-6m5sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:30.024086 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:30.024048    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m5sc\" (UniqueName: \"kubernetes.io/projected/f7e8dda6-863d-4264-966a-a922a45fa930-kube-api-access-6m5sc\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:53:30.024086 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:30.024077    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7e8dda6-863d-4264-966a-a922a45fa930-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:53:30.024086 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:30.024089    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7e8dda6-863d-4264-966a-a922a45fa930-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:53:30.031805 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:30.031776    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p"]
Oct 30 00:53:30.037148 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:30.037095    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3a2d1fa-2hh4p"]
Oct 30 00:53:30.267430 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:30.267334    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b661e154-e205-431d-8835-13678bae8175" path="/var/lib/kubelet/pods/b661e154-e205-431d-8835-13678bae8175/volumes"
Oct 30 00:53:30.673140 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:30.673110    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j"
Oct 30 00:53:30.673140 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:30.673136    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j" event={"ID":"f7e8dda6-863d-4264-966a-a922a45fa930","Type":"ContainerDied","Data":"88f348fbc3dcfdeab9465a8553987e503437d934e946a02786d97552ebde5f24"}
Oct 30 00:53:30.673361 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:30.673179    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88f348fbc3dcfdeab9465a8553987e503437d934e946a02786d97552ebde5f24"
Oct 30 00:53:33.216757 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:33.212086    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:53:36.384730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:36.383821    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:36.607732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:36.596233    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 00:53:36.655836 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:36.642114    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": context deadline exceeded"
Oct 30 00:53:36.753755 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:36.745125    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:36.753755 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:36.752175    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:36.844794 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:36.778914    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 00:53:37.005798 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:37.000050    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 00:53:37.022736 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:37.013981    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 00:53:37.701733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:37.649122    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:37.701733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:37.649880    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:53:37.809729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:37.802970    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:37.809729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:37.803782    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:53:37.809729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:37.804902    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:37.934731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:37.929793    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:37.934731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:37.930317    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:53:38.069478 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:38.069427    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:38.454792 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:38.094050    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 00:53:38.454792 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:38.094090    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded"
Oct 30 00:53:38.454792 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:38.095137    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:38.454792 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:38.105657    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": context deadline exceeded"
Oct 30 00:53:38.454792 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:38.172319    6099 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-tljgj container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.16:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:53:38.454792 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:38.453012    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tljgj" podUID="12bf7b6a-fd62-4921-910d-f9fff2b347d1" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.16:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:38.454792 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:38.190417    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 00:53:38.454792 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:38.172626    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:53:38.454792 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:38.230412    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm" podUID="d94d18a3-84f0-4225-bf91-19ce9a058574" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:38.553565 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:38.553520    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:38.716783 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:38.703836    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:38.722116 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:38.722074    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/health\": context deadline exceeded"
Oct 30 00:53:38.732442 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:38.732406    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Oct 30 00:53:38.741847 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:38.741808    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:38.742270 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:38.742197    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:53:38.836731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:38.825337    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 00:53:39.033789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.009607    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 00:53:39.220728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.191388    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:53:39.221073 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.221048    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:53:39.221239 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.221223    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t"]
Oct 30 00:53:39.221410 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.221397    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:53:39.231215 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.231189    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:53:39.232349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.232326    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds"]
Oct 30 00:53:39.238804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.238785    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:53:39.453587 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.452779    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:39.453587 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.453040    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:39.453587 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.453380    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 00:53:39.453587 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.453437    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:53:39.453587 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.453499    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:39.453587 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.453541    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 00:53:39.691502 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.691472    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:53:39.767816 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.765637    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 00:53:39.821678 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.815972    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:39.822354 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.822065    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:39.822930 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.822555    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:53:39.823320 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.823304    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:53:39.835665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.823211    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:39.836254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:39.836015    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:40.042183 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:40.039875    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:40.454198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:40.454101    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:53:40.456721 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:40.456686    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:53:40.478031 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:40.478007    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:53:40.478285 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:40.478263    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:53:40.491025 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:40.490998    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:53:40.491243 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:40.491230    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:53:40.492676 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:40.491690    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:53:40.492676 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:40.491748    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 00:53:40.492676 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:40.491790    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:53:40.492676 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:40.491822    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:53:40.492676 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:40.491853    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:53:40.664989 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:40.664363    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d"]
Oct 30 00:53:40.683169 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:40.673570    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e708b2f-vw27d"]
Oct 30 00:53:42.274666 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:42.274636    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaae5a9d-72aa-4817-8135-8f6ba657e420" path="/var/lib/kubelet/pods/aaae5a9d-72aa-4817-8135-8f6ba657e420/volumes"
Oct 30 00:53:44.084798 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.084754    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78"]
Oct 30 00:53:44.101527 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.101487    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a21d539b-6qj78"]
Oct 30 00:53:44.101810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.101543    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln"]
Oct 30 00:53:44.113503 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.113471    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74b07396-7ghln"]
Oct 30 00:53:44.266356 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.266317    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t" podUID="3f5d566c-1883-4a76-a994-8686faa32eee" containerName="payload-processing" containerID="cri-o://457d24b1bc5560bdbdebb7785cb4e314138dbc32c0a78fa82961fafa11884ef9" gracePeriod=25
Oct 30 00:53:44.267325 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.267293    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="073ef7a4-c980-4e48-bc70-39903932a68a" path="/var/lib/kubelet/pods/073ef7a4-c980-4e48-bc70-39903932a68a/volumes"
Oct 30 00:53:44.267993 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.267972    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1acb3744-7d81-4e2c-a358-3768bfcbac34" path="/var/lib/kubelet/pods/1acb3744-7d81-4e2c-a358-3768bfcbac34/volumes"
Oct 30 00:53:44.278775 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.278732    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds" podUID="a794233b-4a04-49e2-8dc4-28b167e8cc95" containerName="payload-pre-processing" containerID="cri-o://43e07cd47c06e7d41fda9919dcf56fc9368b2cfc33ef2e57fe6f984fa74bc715" gracePeriod=25
Oct 30 00:53:44.727877 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.727796    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds"
Oct 30 00:53:44.764034 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.764000    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t"
Oct 30 00:53:44.840725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.837331    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5d566c-1883-4a76-a994-8686faa32eee-kube-api-access-qxcfr\" (UniqueName: \"kubernetes.io/projected/3f5d566c-1883-4a76-a994-8686faa32eee-kube-api-access-qxcfr\") pod \"3f5d566c-1883-4a76-a994-8686faa32eee\" (UID: \"3f5d566c-1883-4a76-a994-8686faa32eee\") "
Oct 30 00:53:44.840725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.837451    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f5d566c-1883-4a76-a994-8686faa32eee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5d566c-1883-4a76-a994-8686faa32eee-plugins-config-volume\") pod \"3f5d566c-1883-4a76-a994-8686faa32eee\" (UID: \"3f5d566c-1883-4a76-a994-8686faa32eee\") "
Oct 30 00:53:44.840725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.837558    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a794233b-4a04-49e2-8dc4-28b167e8cc95-kube-api-access-sjqx6\" (UniqueName: \"kubernetes.io/projected/a794233b-4a04-49e2-8dc4-28b167e8cc95-kube-api-access-sjqx6\") pod \"a794233b-4a04-49e2-8dc4-28b167e8cc95\" (UID: \"a794233b-4a04-49e2-8dc4-28b167e8cc95\") "
Oct 30 00:53:44.840725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.837740    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a794233b-4a04-49e2-8dc4-28b167e8cc95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a794233b-4a04-49e2-8dc4-28b167e8cc95-plugins-config-volume\") pod \"a794233b-4a04-49e2-8dc4-28b167e8cc95\" (UID: \"a794233b-4a04-49e2-8dc4-28b167e8cc95\") "
Oct 30 00:53:44.840725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.839164    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a794233b-4a04-49e2-8dc4-28b167e8cc95-plugins-config-volume" pod "a794233b-4a04-49e2-8dc4-28b167e8cc95" (UID: "a794233b-4a04-49e2-8dc4-28b167e8cc95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:44.846873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.841931    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5d566c-1883-4a76-a994-8686faa32eee-kube-api-access-qxcfr" pod "3f5d566c-1883-4a76-a994-8686faa32eee" (UID: "3f5d566c-1883-4a76-a994-8686faa32eee"). InnerVolumeSpecName "kube-api-access-qxcfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:44.846873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.842273    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5d566c-1883-4a76-a994-8686faa32eee-plugins-config-volume" pod "3f5d566c-1883-4a76-a994-8686faa32eee" (UID: "3f5d566c-1883-4a76-a994-8686faa32eee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:44.846873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.844139    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a794233b-4a04-49e2-8dc4-28b167e8cc95-kube-api-access-sjqx6" pod "a794233b-4a04-49e2-8dc4-28b167e8cc95" (UID: "a794233b-4a04-49e2-8dc4-28b167e8cc95"). InnerVolumeSpecName "kube-api-access-sjqx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:44.941681 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.941578    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxcfr\" (UniqueName: \"kubernetes.io/projected/3f5d566c-1883-4a76-a994-8686faa32eee-kube-api-access-qxcfr\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:53:44.941681 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.941618    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5d566c-1883-4a76-a994-8686faa32eee-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:53:44.941681 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.941634    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjqx6\" (UniqueName: \"kubernetes.io/projected/a794233b-4a04-49e2-8dc4-28b167e8cc95-kube-api-access-sjqx6\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:53:44.941681 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:44.941651    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a794233b-4a04-49e2-8dc4-28b167e8cc95-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:53:45.151116 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.151079    6099 generic.go:356] "Generic (PLEG): container finished" podID="3f5d566c-1883-4a76-a994-8686faa32eee" containerID="457d24b1bc5560bdbdebb7785cb4e314138dbc32c0a78fa82961fafa11884ef9" exitCode=0
Oct 30 00:53:45.151585 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.151365    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t"
Oct 30 00:53:45.152057 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.152009    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t" event={"ID":"3f5d566c-1883-4a76-a994-8686faa32eee","Type":"ContainerDied","Data":"457d24b1bc5560bdbdebb7785cb4e314138dbc32c0a78fa82961fafa11884ef9"}
Oct 30 00:53:45.152181 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.152076    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t" event={"ID":"3f5d566c-1883-4a76-a994-8686faa32eee","Type":"ContainerDied","Data":"b9fdb1530c5788fefe43727673fd25ef091c3fe6db72addefb6361f257bfb074"}
Oct 30 00:53:45.152181 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.152111    6099 scope.go:122] "RemoveContainer" containerID="457d24b1bc5560bdbdebb7785cb4e314138dbc32c0a78fa82961fafa11884ef9"
Oct 30 00:53:45.154827 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.154622    6099 generic.go:356] "Generic (PLEG): container finished" podID="a794233b-4a04-49e2-8dc4-28b167e8cc95" containerID="43e07cd47c06e7d41fda9919dcf56fc9368b2cfc33ef2e57fe6f984fa74bc715" exitCode=0
Oct 30 00:53:45.154827 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.154756    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds" event={"ID":"a794233b-4a04-49e2-8dc4-28b167e8cc95","Type":"ContainerDied","Data":"43e07cd47c06e7d41fda9919dcf56fc9368b2cfc33ef2e57fe6f984fa74bc715"}
Oct 30 00:53:45.154827 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.154800    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds" event={"ID":"a794233b-4a04-49e2-8dc4-28b167e8cc95","Type":"ContainerDied","Data":"78917c44bcdb22d8a6292f752bd4fb1a23e1cbb4a12214c9040da4fbca180b98"}
Oct 30 00:53:45.155057 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.154876    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds"
Oct 30 00:53:45.181251 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.181223    6099 scope.go:122] "RemoveContainer" containerID="457d24b1bc5560bdbdebb7785cb4e314138dbc32c0a78fa82961fafa11884ef9"
Oct 30 00:53:45.181603 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:53:45.181584    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457d24b1bc5560bdbdebb7785cb4e314138dbc32c0a78fa82961fafa11884ef9\": container with ID starting with 457d24b1bc5560bdbdebb7785cb4e314138dbc32c0a78fa82961fafa11884ef9 not found: ID does not exist" containerID="457d24b1bc5560bdbdebb7785cb4e314138dbc32c0a78fa82961fafa11884ef9"
Oct 30 00:53:45.181686 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.181618    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457d24b1bc5560bdbdebb7785cb4e314138dbc32c0a78fa82961fafa11884ef9"} err="failed to get container status \"457d24b1bc5560bdbdebb7785cb4e314138dbc32c0a78fa82961fafa11884ef9\": rpc error: code = NotFound desc = could not find container \"457d24b1bc5560bdbdebb7785cb4e314138dbc32c0a78fa82961fafa11884ef9\": container with ID starting with 457d24b1bc5560bdbdebb7785cb4e314138dbc32c0a78fa82961fafa11884ef9 not found: ID does not exist"
Oct 30 00:53:45.181686 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.181645    6099 scope.go:122] "RemoveContainer" containerID="43e07cd47c06e7d41fda9919dcf56fc9368b2cfc33ef2e57fe6f984fa74bc715"
Oct 30 00:53:45.195337 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.195317    6099 scope.go:122] "RemoveContainer" containerID="43e07cd47c06e7d41fda9919dcf56fc9368b2cfc33ef2e57fe6f984fa74bc715"
Oct 30 00:53:45.195627 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:53:45.195600    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e07cd47c06e7d41fda9919dcf56fc9368b2cfc33ef2e57fe6f984fa74bc715\": container with ID starting with 43e07cd47c06e7d41fda9919dcf56fc9368b2cfc33ef2e57fe6f984fa74bc715 not found: ID does not exist" containerID="43e07cd47c06e7d41fda9919dcf56fc9368b2cfc33ef2e57fe6f984fa74bc715"
Oct 30 00:53:45.195768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.195633    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e07cd47c06e7d41fda9919dcf56fc9368b2cfc33ef2e57fe6f984fa74bc715"} err="failed to get container status \"43e07cd47c06e7d41fda9919dcf56fc9368b2cfc33ef2e57fe6f984fa74bc715\": rpc error: code = NotFound desc = could not find container \"43e07cd47c06e7d41fda9919dcf56fc9368b2cfc33ef2e57fe6f984fa74bc715\": container with ID starting with 43e07cd47c06e7d41fda9919dcf56fc9368b2cfc33ef2e57fe6f984fa74bc715 not found: ID does not exist"
Oct 30 00:53:45.229779 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.224810    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds"]
Oct 30 00:53:45.237074 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.237045    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c4e6c558-5f57cf845f-82kds"]
Oct 30 00:53:45.319789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.319755    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t"]
Oct 30 00:53:45.325903 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.325870    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c4e6c558-5f756664fb-gsb6t"]
Oct 30 00:53:45.357882 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.357851    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j"]
Oct 30 00:53:45.374212 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:45.374182    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c4e6c558-fcv5j"]
Oct 30 00:53:46.263409 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:46.263374    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f5d566c-1883-4a76-a994-8686faa32eee" path="/var/lib/kubelet/pods/3f5d566c-1883-4a76-a994-8686faa32eee/volumes"
Oct 30 00:53:46.263968 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:46.263950    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a794233b-4a04-49e2-8dc4-28b167e8cc95" path="/var/lib/kubelet/pods/a794233b-4a04-49e2-8dc4-28b167e8cc95/volumes"
Oct 30 00:53:46.264499 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:46.264482    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7e8dda6-863d-4264-966a-a922a45fa930" path="/var/lib/kubelet/pods/f7e8dda6-863d-4264-966a-a922a45fa930/volumes"
Oct 30 00:53:46.635397 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:53:46.635354    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:53:47.656740 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.656607    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s"]
Oct 30 00:53:47.657846 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.657789    6099 cpu_manager.go:414] "containerMap: removing container" podUID="a794233b-4a04-49e2-8dc4-28b167e8cc95" containerName="payload-pre-processing"
Oct 30 00:53:47.657846 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.657815    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a794233b-4a04-49e2-8dc4-28b167e8cc95" containerName="payload-pre-processing"
Oct 30 00:53:47.658041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.657882    6099 cpu_manager.go:414] "containerMap: removing container" podUID="3f5d566c-1883-4a76-a994-8686faa32eee" containerName="payload-processing"
Oct 30 00:53:47.658041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.657890    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5d566c-1883-4a76-a994-8686faa32eee" containerName="payload-processing"
Oct 30 00:53:47.658041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.657928    6099 cpu_manager.go:414] "containerMap: removing container" podUID="f7e8dda6-863d-4264-966a-a922a45fa930" containerName="revoke-keys"
Oct 30 00:53:47.658041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.657937    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7e8dda6-863d-4264-966a-a922a45fa930" containerName="revoke-keys"
Oct 30 00:53:47.658308 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.658111    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5d566c-1883-4a76-a994-8686faa32eee" containerName="payload-processing"
Oct 30 00:53:47.658308 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.658129    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7e8dda6-863d-4264-966a-a922a45fa930" containerName="revoke-keys"
Oct 30 00:53:47.658308 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.658142    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a794233b-4a04-49e2-8dc4-28b167e8cc95" containerName="payload-pre-processing"
Oct 30 00:53:47.665091 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.665057    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s"
Oct 30 00:53:47.676285 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.676245    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s"]
Oct 30 00:53:47.680733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.680694    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:53:47.694300 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.692104    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:53:47.694300 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.692225    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:53:47.694300 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.692375    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:53:47.721727 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.701988    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40630740-9e4a-49cd-8aac-f4fcf9e4b378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c07489-s4x7s\" (UID: \"40630740-9e4a-49cd-8aac-f4fcf9e4b378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s"
Oct 30 00:53:47.721727 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.702109    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40630740-9e4a-49cd-8aac-f4fcf9e4b378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c07489-s4x7s\" (UID: \"40630740-9e4a-49cd-8aac-f4fcf9e4b378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s"
Oct 30 00:53:47.721727 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.702237    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-787k2\" (UniqueName: \"kubernetes.io/projected/40630740-9e4a-49cd-8aac-f4fcf9e4b378-kube-api-access-787k2\") pod \"maas-api-revoke-sub-a9c07489-s4x7s\" (UID: \"40630740-9e4a-49cd-8aac-f4fcf9e4b378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s"
Oct 30 00:53:47.803795 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.803739    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40630740-9e4a-49cd-8aac-f4fcf9e4b378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c07489-s4x7s\" (UID: \"40630740-9e4a-49cd-8aac-f4fcf9e4b378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s"
Oct 30 00:53:47.804006 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.803883    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40630740-9e4a-49cd-8aac-f4fcf9e4b378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c07489-s4x7s\" (UID: \"40630740-9e4a-49cd-8aac-f4fcf9e4b378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s"
Oct 30 00:53:47.804098 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.804007    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-787k2\" (UniqueName: \"kubernetes.io/projected/40630740-9e4a-49cd-8aac-f4fcf9e4b378-kube-api-access-787k2\") pod \"maas-api-revoke-sub-a9c07489-s4x7s\" (UID: \"40630740-9e4a-49cd-8aac-f4fcf9e4b378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s"
Oct 30 00:53:47.804671 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.804644    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40630740-9e4a-49cd-8aac-f4fcf9e4b378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c07489-s4x7s\" (UID: \"40630740-9e4a-49cd-8aac-f4fcf9e4b378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s"
Oct 30 00:53:47.806373 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.806346    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40630740-9e4a-49cd-8aac-f4fcf9e4b378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c07489-s4x7s\" (UID: \"40630740-9e4a-49cd-8aac-f4fcf9e4b378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s"
Oct 30 00:53:47.826771 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.826741    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-787k2\" (UniqueName: \"kubernetes.io/projected/40630740-9e4a-49cd-8aac-f4fcf9e4b378-kube-api-access-787k2\") pod \"maas-api-revoke-sub-a9c07489-s4x7s\" (UID: \"40630740-9e4a-49cd-8aac-f4fcf9e4b378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s"
Oct 30 00:53:47.992505 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:47.992436    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s"
Oct 30 00:53:48.327569 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:53:48.327537    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40630740_9e4a_49cd_8aac_f4fcf9e4b378.slice/crio-6302fd2018ec760d2a4f1e169b6675aae85a7b3f2f006a08cb54f5c4a396f9a8 WatchSource:0}: Error finding container 6302fd2018ec760d2a4f1e169b6675aae85a7b3f2f006a08cb54f5c4a396f9a8: Status 404 returned error can't find the container with id 6302fd2018ec760d2a4f1e169b6675aae85a7b3f2f006a08cb54f5c4a396f9a8
Oct 30 00:53:48.330233 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:48.330162    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s"]
Oct 30 00:53:49.202055 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:49.202010    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s" event={"ID":"40630740-9e4a-49cd-8aac-f4fcf9e4b378","Type":"ContainerStarted","Data":"0d1258dbc93773aafde5a9e5e4c3736c4944d44a98121352bf1d7feb62951d87"}
Oct 30 00:53:49.202536 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:49.202110    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s" event={"ID":"40630740-9e4a-49cd-8aac-f4fcf9e4b378","Type":"ContainerStarted","Data":"6302fd2018ec760d2a4f1e169b6675aae85a7b3f2f006a08cb54f5c4a396f9a8"}
Oct 30 00:53:49.246343 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:49.246292    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s" podStartSLOduration=2.246278075 podStartE2EDuration="2.246278075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:49.244419048 +0000 UTC m=+1123.101326813" watchObservedRunningTime="2026-10-30 00:53:49.246278075 +0000 UTC m=+1123.103185830"
Oct 30 00:53:50.213066 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:50.213033    6099 generic.go:356] "Generic (PLEG): container finished" podID="40630740-9e4a-49cd-8aac-f4fcf9e4b378" containerID="0d1258dbc93773aafde5a9e5e4c3736c4944d44a98121352bf1d7feb62951d87" exitCode=0
Oct 30 00:53:50.213479 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:50.213122    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s" event={"ID":"40630740-9e4a-49cd-8aac-f4fcf9e4b378","Type":"ContainerDied","Data":"0d1258dbc93773aafde5a9e5e4c3736c4944d44a98121352bf1d7feb62951d87"}
Oct 30 00:53:51.412078 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.412037    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g"]
Oct 30 00:53:51.424301 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.421129    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g"
Oct 30 00:53:51.424301 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.422553    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g"]
Oct 30 00:53:51.449641 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.449610    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-573837e3-dockercfg-qv8f8\""
Oct 30 00:53:51.449641 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.449636    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-573837e3\""
Oct 30 00:53:51.526419 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.526368    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dkft\" (UniqueName: \"kubernetes.io/projected/8c07f884-b799-4e04-9e65-d59174993df5-kube-api-access-2dkft\") pod \"payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g\" (UID: \"8c07f884-b799-4e04-9e65-d59174993df5\") " pod="openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g"
Oct 30 00:53:51.526677 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.526656    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c07f884-b799-4e04-9e65-d59174993df5-plugins-config-volume\") pod \"payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g\" (UID: \"8c07f884-b799-4e04-9e65-d59174993df5\") " pod="openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g"
Oct 30 00:53:51.614387 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.614356    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s"
Oct 30 00:53:51.628883 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.628818    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dkft\" (UniqueName: \"kubernetes.io/projected/8c07f884-b799-4e04-9e65-d59174993df5-kube-api-access-2dkft\") pod \"payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g\" (UID: \"8c07f884-b799-4e04-9e65-d59174993df5\") " pod="openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g"
Oct 30 00:53:51.629019 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.628974    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c07f884-b799-4e04-9e65-d59174993df5-plugins-config-volume\") pod \"payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g\" (UID: \"8c07f884-b799-4e04-9e65-d59174993df5\") " pod="openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g"
Oct 30 00:53:51.629628 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.629608    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c07f884-b799-4e04-9e65-d59174993df5-plugins-config-volume\") pod \"payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g\" (UID: \"8c07f884-b799-4e04-9e65-d59174993df5\") " pod="openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g"
Oct 30 00:53:51.652064 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.652041    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dkft\" (UniqueName: \"kubernetes.io/projected/8c07f884-b799-4e04-9e65-d59174993df5-kube-api-access-2dkft\") pod \"payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g\" (UID: \"8c07f884-b799-4e04-9e65-d59174993df5\") " pod="openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g"
Oct 30 00:53:51.730597 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.730567    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40630740-9e4a-49cd-8aac-f4fcf9e4b378-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40630740-9e4a-49cd-8aac-f4fcf9e4b378-maas-api-xks-ca\") pod \"40630740-9e4a-49cd-8aac-f4fcf9e4b378\" (UID: \"40630740-9e4a-49cd-8aac-f4fcf9e4b378\") "
Oct 30 00:53:51.730743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.730651    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40630740-9e4a-49cd-8aac-f4fcf9e4b378-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40630740-9e4a-49cd-8aac-f4fcf9e4b378-maas-api-openshift-ca\") pod \"40630740-9e4a-49cd-8aac-f4fcf9e4b378\" (UID: \"40630740-9e4a-49cd-8aac-f4fcf9e4b378\") "
Oct 30 00:53:51.730800 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.730741    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40630740-9e4a-49cd-8aac-f4fcf9e4b378-kube-api-access-787k2\" (UniqueName: \"kubernetes.io/projected/40630740-9e4a-49cd-8aac-f4fcf9e4b378-kube-api-access-787k2\") pod \"40630740-9e4a-49cd-8aac-f4fcf9e4b378\" (UID: \"40630740-9e4a-49cd-8aac-f4fcf9e4b378\") "
Oct 30 00:53:51.731004 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.730977    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40630740-9e4a-49cd-8aac-f4fcf9e4b378-maas-api-openshift-ca" pod "40630740-9e4a-49cd-8aac-f4fcf9e4b378" (UID: "40630740-9e4a-49cd-8aac-f4fcf9e4b378"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:51.731450 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.731431    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40630740-9e4a-49cd-8aac-f4fcf9e4b378-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:53:51.732693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.732670    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40630740-9e4a-49cd-8aac-f4fcf9e4b378-maas-api-xks-ca" pod "40630740-9e4a-49cd-8aac-f4fcf9e4b378" (UID: "40630740-9e4a-49cd-8aac-f4fcf9e4b378"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:53:51.732972 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.732953    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40630740-9e4a-49cd-8aac-f4fcf9e4b378-kube-api-access-787k2" pod "40630740-9e4a-49cd-8aac-f4fcf9e4b378" (UID: "40630740-9e4a-49cd-8aac-f4fcf9e4b378"). InnerVolumeSpecName "kube-api-access-787k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:51.747170 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.747147    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g"
Oct 30 00:53:51.837397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.837362    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40630740-9e4a-49cd-8aac-f4fcf9e4b378-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:53:51.837397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:51.837398    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-787k2\" (UniqueName: \"kubernetes.io/projected/40630740-9e4a-49cd-8aac-f4fcf9e4b378-kube-api-access-787k2\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:53:52.057622 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:53:52.057587    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c07f884_b799_4e04_9e65_d59174993df5.slice/crio-bd2b1243ed1091db24858ffa20df02170adc57c1493d4781f588e23f0940f2ba WatchSource:0}: Error finding container bd2b1243ed1091db24858ffa20df02170adc57c1493d4781f588e23f0940f2ba: Status 404 returned error can't find the container with id bd2b1243ed1091db24858ffa20df02170adc57c1493d4781f588e23f0940f2ba
Oct 30 00:53:52.060289 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:52.060264    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g"]
Oct 30 00:53:52.365343 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:52.365253    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g" event={"ID":"8c07f884-b799-4e04-9e65-d59174993df5","Type":"ContainerStarted","Data":"e7823e0f7d4a33625442ae267b6840eecb26faea0110a7d22f99d00f4ffba289"}
Oct 30 00:53:52.365343 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:52.365316    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g"
Oct 30 00:53:52.365343 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:52.365340    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g" event={"ID":"8c07f884-b799-4e04-9e65-d59174993df5","Type":"ContainerStarted","Data":"bd2b1243ed1091db24858ffa20df02170adc57c1493d4781f588e23f0940f2ba"}
Oct 30 00:53:52.366864 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:52.366843    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s"
Oct 30 00:53:52.366991 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:52.366890    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s" event={"ID":"40630740-9e4a-49cd-8aac-f4fcf9e4b378","Type":"ContainerDied","Data":"6302fd2018ec760d2a4f1e169b6675aae85a7b3f2f006a08cb54f5c4a396f9a8"}
Oct 30 00:53:52.366991 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:52.366939    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6302fd2018ec760d2a4f1e169b6675aae85a7b3f2f006a08cb54f5c4a396f9a8"
Oct 30 00:53:52.405594 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:53:52.405542    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g" podStartSLOduration=1.405527691 podStartE2EDuration="1.405527691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:52.404130505 +0000 UTC m=+1126.261038272" watchObservedRunningTime="2026-10-30 00:53:52.405527691 +0000 UTC m=+1126.262435445"
Oct 30 00:54:03.404930 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:03.404896    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g"
Oct 30 00:54:08.631696 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:08.631663    6099 scope.go:122] "RemoveContainer" containerID="c7385e4f2e7ae265292b256134aac379bad047675a5c9bf01c81ba2914b6e2c7"
Oct 30 00:54:08.643126 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:08.643106    6099 scope.go:122] "RemoveContainer" containerID="7513319f9ee01cccb7a3179c08b1a765334a874466df6c5fdd7e2bc8cf47e297"
Oct 30 00:54:08.666243 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:08.666216    6099 scope.go:122] "RemoveContainer" containerID="dd1dcde3c069917f7d8e09267dc913314057f20512efd3b346c934a52f63a6c2"
Oct 30 00:54:09.628870 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:09.628834    6099 generic.go:356] "Generic (PLEG): container finished" podID="1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" containerID="2aab72d2a83ca606bff17e89d4f1ce26a87b69cd92b97c409d5e9eed830d959c" exitCode=0
Oct 30 00:54:09.629127 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:09.628904    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx" event={"ID":"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6","Type":"ContainerDied","Data":"2aab72d2a83ca606bff17e89d4f1ce26a87b69cd92b97c409d5e9eed830d959c"}
Oct 30 00:54:09.629352 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:09.629337    6099 scope.go:122] "RemoveContainer" containerID="2aab72d2a83ca606bff17e89d4f1ce26a87b69cd92b97c409d5e9eed830d959c"
Oct 30 00:54:09.630789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:09.630759    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:54:09.630873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:09.630826    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:54:09.630911 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:09.630875    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:54:10.640378 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:10.640329    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx" event={"ID":"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6","Type":"ContainerStarted","Data":"ca0f9be4df1c8cc2e190e17c568dad6b0747a4c6e8019294978af8d851d649c9"}
Oct 30 00:54:11.397429 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:11.397395    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:54:11.398667 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:11.398643    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 00:54:11.398845 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:11.398687    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx" podUID="1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 00:54:12.043217 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:12.043188    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:54:12.146033 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:12.146000    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95"]
Oct 30 00:54:12.146796 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:12.146769    6099 cpu_manager.go:414] "containerMap: removing container" podUID="40630740-9e4a-49cd-8aac-f4fcf9e4b378" containerName="revoke-keys"
Oct 30 00:54:12.146796 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:12.146797    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40630740-9e4a-49cd-8aac-f4fcf9e4b378" containerName="revoke-keys"
Oct 30 00:54:12.146934 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:12.146924    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="40630740-9e4a-49cd-8aac-f4fcf9e4b378" containerName="revoke-keys"
Oct 30 00:54:12.150172 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:12.150155    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95"
Oct 30 00:54:12.180931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:12.180901    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-60e8f4\""
Oct 30 00:54:12.181296 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:12.181274    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-60e8f4-dockercfg-2px8s\""
Oct 30 00:54:12.201807 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:12.201767    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95"]
Oct 30 00:54:12.353777 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:12.353378    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41feca8d-0a29-42e4-9969-c1f6e4ee7f36-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95\" (UID: \"41feca8d-0a29-42e4-9969-c1f6e4ee7f36\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95"
Oct 30 00:54:12.363761 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:12.354044    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n689r\" (UniqueName: \"kubernetes.io/projected/41feca8d-0a29-42e4-9969-c1f6e4ee7f36-kube-api-access-n689r\") pod \"payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95\" (UID: \"41feca8d-0a29-42e4-9969-c1f6e4ee7f36\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95"
Oct 30 00:54:12.412731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:12.404086    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 00:54:12.412731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:12.404162    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx" podUID="1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 00:54:12.470788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:12.463306    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n689r\" (UniqueName: \"kubernetes.io/projected/41feca8d-0a29-42e4-9969-c1f6e4ee7f36-kube-api-access-n689r\") pod \"payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95\" (UID: \"41feca8d-0a29-42e4-9969-c1f6e4ee7f36\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95"
Oct 30 00:54:12.470788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:12.463379    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41feca8d-0a29-42e4-9969-c1f6e4ee7f36-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95\" (UID: \"41feca8d-0a29-42e4-9969-c1f6e4ee7f36\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95"
Oct 30 00:54:12.470788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:12.465348    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41feca8d-0a29-42e4-9969-c1f6e4ee7f36-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95\" (UID: \"41feca8d-0a29-42e4-9969-c1f6e4ee7f36\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95"
Oct 30 00:54:12.529737 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:12.521522    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n689r\" (UniqueName: \"kubernetes.io/projected/41feca8d-0a29-42e4-9969-c1f6e4ee7f36-kube-api-access-n689r\") pod \"payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95\" (UID: \"41feca8d-0a29-42e4-9969-c1f6e4ee7f36\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95"
Oct 30 00:54:12.763277 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:12.763193    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95"
Oct 30 00:54:13.154259 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:54:13.154225    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41feca8d_0a29_42e4_9969_c1f6e4ee7f36.slice/crio-99cfff83f6db7556d932c2fc873483b1b685878149398109fb5f2d2584f7570d WatchSource:0}: Error finding container 99cfff83f6db7556d932c2fc873483b1b685878149398109fb5f2d2584f7570d: Status 404 returned error can't find the container with id 99cfff83f6db7556d932c2fc873483b1b685878149398109fb5f2d2584f7570d
Oct 30 00:54:13.154608 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:13.154400    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95"]
Oct 30 00:54:13.398152 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:13.398116    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 00:54:13.398321 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:13.398164    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx" podUID="1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 00:54:13.699478 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:13.699442    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95" event={"ID":"41feca8d-0a29-42e4-9969-c1f6e4ee7f36","Type":"ContainerStarted","Data":"9f63313577e881b892d25c7b079c4f25dec08298c35f2a6750860351198f2eda"}
Oct 30 00:54:13.699665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:13.699532    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95" event={"ID":"41feca8d-0a29-42e4-9969-c1f6e4ee7f36","Type":"ContainerStarted","Data":"99cfff83f6db7556d932c2fc873483b1b685878149398109fb5f2d2584f7570d"}
Oct 30 00:54:13.699840 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:13.699681    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95"
Oct 30 00:54:13.741754 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:13.741691    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95" podStartSLOduration=1.741678512 podStartE2EDuration="1.741678512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:13.738632834 +0000 UTC m=+1147.595540590" watchObservedRunningTime="2026-10-30 00:54:13.741678512 +0000 UTC m=+1147.598586328"
Oct 30 00:54:14.398268 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:14.398188    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 00:54:14.398268 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:14.398248    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx" podUID="1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 00:54:15.397502 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:15.397460    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 00:54:15.397694 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:15.397521    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx" podUID="1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 00:54:16.403915 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:16.403882    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:54:16.405557 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:16.405530    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 00:54:16.639895 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:54:16.639857    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:54:23.867095 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:23.867061    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj"]
Oct 30 00:54:23.871335 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:23.871312    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj"
Oct 30 00:54:23.891123 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:23.891096    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj"]
Oct 30 00:54:23.895934 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:23.895912    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:54:23.896040 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:23.895948    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:54:23.896040 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:23.895961    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:54:23.897347 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:23.897330    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:54:23.963813 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:23.963781    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2nxr\" (UniqueName: \"kubernetes.io/projected/63d33f44-8a31-4aa9-b169-3d1f819a0335-kube-api-access-f2nxr\") pod \"maas-api-revoke-sub-06521758-czhvj\" (UID: \"63d33f44-8a31-4aa9-b169-3d1f819a0335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj"
Oct 30 00:54:23.963951 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:23.963830    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d33f44-8a31-4aa9-b169-3d1f819a0335-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06521758-czhvj\" (UID: \"63d33f44-8a31-4aa9-b169-3d1f819a0335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj"
Oct 30 00:54:23.964061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:23.964038    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d33f44-8a31-4aa9-b169-3d1f819a0335-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06521758-czhvj\" (UID: \"63d33f44-8a31-4aa9-b169-3d1f819a0335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj"
Oct 30 00:54:24.066280 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:24.066244    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d33f44-8a31-4aa9-b169-3d1f819a0335-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06521758-czhvj\" (UID: \"63d33f44-8a31-4aa9-b169-3d1f819a0335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj"
Oct 30 00:54:24.066514 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:24.066494    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2nxr\" (UniqueName: \"kubernetes.io/projected/63d33f44-8a31-4aa9-b169-3d1f819a0335-kube-api-access-f2nxr\") pod \"maas-api-revoke-sub-06521758-czhvj\" (UID: \"63d33f44-8a31-4aa9-b169-3d1f819a0335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj"
Oct 30 00:54:24.066565 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:24.066549    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d33f44-8a31-4aa9-b169-3d1f819a0335-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06521758-czhvj\" (UID: \"63d33f44-8a31-4aa9-b169-3d1f819a0335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj"
Oct 30 00:54:24.067638 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:24.067612    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d33f44-8a31-4aa9-b169-3d1f819a0335-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06521758-czhvj\" (UID: \"63d33f44-8a31-4aa9-b169-3d1f819a0335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj"
Oct 30 00:54:24.069625 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:24.069603    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d33f44-8a31-4aa9-b169-3d1f819a0335-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06521758-czhvj\" (UID: \"63d33f44-8a31-4aa9-b169-3d1f819a0335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj"
Oct 30 00:54:24.092405 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:24.092377    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2nxr\" (UniqueName: \"kubernetes.io/projected/63d33f44-8a31-4aa9-b169-3d1f819a0335-kube-api-access-f2nxr\") pod \"maas-api-revoke-sub-06521758-czhvj\" (UID: \"63d33f44-8a31-4aa9-b169-3d1f819a0335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj"
Oct 30 00:54:24.186061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:24.185902    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj"
Oct 30 00:54:24.507017 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:54:24.506986    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d33f44_8a31_4aa9_b169_3d1f819a0335.slice/crio-3f41f7b2ab0df7a3f287b3a348b60843187fea997b1b0c99827016c69fe106c0 WatchSource:0}: Error finding container 3f41f7b2ab0df7a3f287b3a348b60843187fea997b1b0c99827016c69fe106c0: Status 404 returned error can't find the container with id 3f41f7b2ab0df7a3f287b3a348b60843187fea997b1b0c99827016c69fe106c0
Oct 30 00:54:24.508935 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:24.508910    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj"]
Oct 30 00:54:24.737451 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:24.737422    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95"
Oct 30 00:54:24.829729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:24.829640    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj" event={"ID":"63d33f44-8a31-4aa9-b169-3d1f819a0335","Type":"ContainerStarted","Data":"b6f56225a0ad2cf1b439e3315559a3a236027e8cf7d1212232795fe5ceb326bd"}
Oct 30 00:54:24.829899 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:24.829745    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj" event={"ID":"63d33f44-8a31-4aa9-b169-3d1f819a0335","Type":"ContainerStarted","Data":"3f41f7b2ab0df7a3f287b3a348b60843187fea997b1b0c99827016c69fe106c0"}
Oct 30 00:54:24.874433 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:24.874365    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj" podStartSLOduration=1.874345677 podStartE2EDuration="1.874345677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:24.868606896 +0000 UTC m=+1158.725514653" watchObservedRunningTime="2026-10-30 00:54:24.874345677 +0000 UTC m=+1158.731253433"
Oct 30 00:54:24.904325 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:24.904292    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:54:25.922720 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:25.922666    6099 generic.go:356] "Generic (PLEG): container finished" podID="63d33f44-8a31-4aa9-b169-3d1f819a0335" containerID="b6f56225a0ad2cf1b439e3315559a3a236027e8cf7d1212232795fe5ceb326bd" exitCode=0
Oct 30 00:54:25.923493 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:25.923463    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj" event={"ID":"63d33f44-8a31-4aa9-b169-3d1f819a0335","Type":"ContainerDied","Data":"b6f56225a0ad2cf1b439e3315559a3a236027e8cf7d1212232795fe5ceb326bd"}
Oct 30 00:54:26.034056 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:26.033998    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt"]
Oct 30 00:54:26.041735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:26.041684    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b93bb10-gprtt"]
Oct 30 00:54:26.264417 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:26.264342    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c3ac3dc-adee-4991-949c-824aefcb7a28" path="/var/lib/kubelet/pods/4c3ac3dc-adee-4991-949c-824aefcb7a28/volumes"
Oct 30 00:54:27.041457 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:27.041418    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf"]
Oct 30 00:54:27.049499 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:27.049469    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b8bd8a4-wx6rf"]
Oct 30 00:54:27.106998 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:27.106975    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj"
Oct 30 00:54:27.250112 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:27.250032    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d33f44-8a31-4aa9-b169-3d1f819a0335-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d33f44-8a31-4aa9-b169-3d1f819a0335-maas-api-xks-ca\") pod \"63d33f44-8a31-4aa9-b169-3d1f819a0335\" (UID: \"63d33f44-8a31-4aa9-b169-3d1f819a0335\") "
Oct 30 00:54:27.250260 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:27.250246    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d33f44-8a31-4aa9-b169-3d1f819a0335-kube-api-access-f2nxr\" (UniqueName: \"kubernetes.io/projected/63d33f44-8a31-4aa9-b169-3d1f819a0335-kube-api-access-f2nxr\") pod \"63d33f44-8a31-4aa9-b169-3d1f819a0335\" (UID: \"63d33f44-8a31-4aa9-b169-3d1f819a0335\") "
Oct 30 00:54:27.250480 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:27.250455    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d33f44-8a31-4aa9-b169-3d1f819a0335-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d33f44-8a31-4aa9-b169-3d1f819a0335-maas-api-openshift-ca\") pod \"63d33f44-8a31-4aa9-b169-3d1f819a0335\" (UID: \"63d33f44-8a31-4aa9-b169-3d1f819a0335\") "
Oct 30 00:54:27.250815 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:27.250794    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d33f44-8a31-4aa9-b169-3d1f819a0335-maas-api-openshift-ca" pod "63d33f44-8a31-4aa9-b169-3d1f819a0335" (UID: "63d33f44-8a31-4aa9-b169-3d1f819a0335"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:54:27.251218 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:27.251200    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d33f44-8a31-4aa9-b169-3d1f819a0335-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:54:27.252627 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:27.252604    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d33f44-8a31-4aa9-b169-3d1f819a0335-maas-api-xks-ca" pod "63d33f44-8a31-4aa9-b169-3d1f819a0335" (UID: "63d33f44-8a31-4aa9-b169-3d1f819a0335"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:54:27.252746 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:27.252662    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d33f44-8a31-4aa9-b169-3d1f819a0335-kube-api-access-f2nxr" pod "63d33f44-8a31-4aa9-b169-3d1f819a0335" (UID: "63d33f44-8a31-4aa9-b169-3d1f819a0335"). InnerVolumeSpecName "kube-api-access-f2nxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:54:27.352620 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:27.352587    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2nxr\" (UniqueName: \"kubernetes.io/projected/63d33f44-8a31-4aa9-b169-3d1f819a0335-kube-api-access-f2nxr\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:54:27.352620 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:27.352620    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d33f44-8a31-4aa9-b169-3d1f819a0335-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:54:27.944376 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:27.944354    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj"
Oct 30 00:54:27.944546 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:27.944392    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj" event={"ID":"63d33f44-8a31-4aa9-b169-3d1f819a0335","Type":"ContainerDied","Data":"3f41f7b2ab0df7a3f287b3a348b60843187fea997b1b0c99827016c69fe106c0"}
Oct 30 00:54:27.944546 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:27.944434    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f41f7b2ab0df7a3f287b3a348b60843187fea997b1b0c99827016c69fe106c0"
Oct 30 00:54:28.263685 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:28.263606    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f065b6-13ab-455d-a287-2e2a1fedaa5e" path="/var/lib/kubelet/pods/f7f065b6-13ab-455d-a287-2e2a1fedaa5e/volumes"
Oct 30 00:54:28.677467 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:28.677435    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6"]
Oct 30 00:54:28.678258 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:28.678236    6099 cpu_manager.go:414] "containerMap: removing container" podUID="63d33f44-8a31-4aa9-b169-3d1f819a0335" containerName="revoke-keys"
Oct 30 00:54:28.678258 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:28.678257    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d33f44-8a31-4aa9-b169-3d1f819a0335" containerName="revoke-keys"
Oct 30 00:54:28.678494 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:28.678479    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d33f44-8a31-4aa9-b169-3d1f819a0335" containerName="revoke-keys"
Oct 30 00:54:28.681509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:28.681488    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6"
Oct 30 00:54:28.706472 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:28.706438    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6"]
Oct 30 00:54:28.709867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:28.707546    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d8090513-dockercfg-5n5zl\""
Oct 30 00:54:28.709867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:28.707598    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d8090513\""
Oct 30 00:54:28.784171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:28.784130    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d97ad33-c251-46b6-b0dd-90112f148297-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6\" (UID: \"4d97ad33-c251-46b6-b0dd-90112f148297\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6"
Oct 30 00:54:28.784624 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:28.784606    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrlbx\" (UniqueName: \"kubernetes.io/projected/4d97ad33-c251-46b6-b0dd-90112f148297-kube-api-access-xrlbx\") pod \"payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6\" (UID: \"4d97ad33-c251-46b6-b0dd-90112f148297\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6"
Oct 30 00:54:28.953838 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:28.950735    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrlbx\" (UniqueName: \"kubernetes.io/projected/4d97ad33-c251-46b6-b0dd-90112f148297-kube-api-access-xrlbx\") pod \"payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6\" (UID: \"4d97ad33-c251-46b6-b0dd-90112f148297\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6"
Oct 30 00:54:28.953838 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:28.951268    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d97ad33-c251-46b6-b0dd-90112f148297-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6\" (UID: \"4d97ad33-c251-46b6-b0dd-90112f148297\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6"
Oct 30 00:54:28.953838 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:28.952468    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d97ad33-c251-46b6-b0dd-90112f148297-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6\" (UID: \"4d97ad33-c251-46b6-b0dd-90112f148297\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6"
Oct 30 00:54:28.990731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:28.988451    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrlbx\" (UniqueName: \"kubernetes.io/projected/4d97ad33-c251-46b6-b0dd-90112f148297-kube-api-access-xrlbx\") pod \"payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6\" (UID: \"4d97ad33-c251-46b6-b0dd-90112f148297\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6"
Oct 30 00:54:28.999083 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:28.999047    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6"
Oct 30 00:54:29.318594 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:54:29.318563    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d97ad33_c251_46b6_b0dd_90112f148297.slice/crio-cc9605c85c338160fe3eeda428ab3c71e35a49c9d3d413e9ff115bf40f496640 WatchSource:0}: Error finding container cc9605c85c338160fe3eeda428ab3c71e35a49c9d3d413e9ff115bf40f496640: Status 404 returned error can't find the container with id cc9605c85c338160fe3eeda428ab3c71e35a49c9d3d413e9ff115bf40f496640
Oct 30 00:54:29.321655 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:29.321631    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6"]
Oct 30 00:54:29.977941 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:29.977900    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6" event={"ID":"4d97ad33-c251-46b6-b0dd-90112f148297","Type":"ContainerStarted","Data":"847e6d93e2aef370f3525898a74ba98413587954f46b76cc9a44f7608f0874c4"}
Oct 30 00:54:29.978136 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:29.977961    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6" event={"ID":"4d97ad33-c251-46b6-b0dd-90112f148297","Type":"ContainerStarted","Data":"cc9605c85c338160fe3eeda428ab3c71e35a49c9d3d413e9ff115bf40f496640"}
Oct 30 00:54:29.978136 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:29.977991    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6"
Oct 30 00:54:30.019786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:30.019733    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6" podStartSLOduration=2.019718846 podStartE2EDuration="2.019718846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:30.017795393 +0000 UTC m=+1163.874703149" watchObservedRunningTime="2026-10-30 00:54:30.019718846 +0000 UTC m=+1163.876626620"
Oct 30 00:54:30.998346 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:30.998300    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z"]
Oct 30 00:54:31.002502 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:31.002482    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z"
Oct 30 00:54:31.023984 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:31.023928    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z"]
Oct 30 00:54:31.027589 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:31.027564    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:54:31.027792 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:31.027566    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:54:31.027792 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:31.027650    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:54:31.028360 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:31.028343    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:54:31.098106 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:31.098066    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-573837e3-8z48z\" (UID: \"d69cf7a3-856b-4cb8-aa7a-f978b93b14f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z"
Oct 30 00:54:31.098279 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:31.098124    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdhcr\" (UniqueName: \"kubernetes.io/projected/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-kube-api-access-tdhcr\") pod \"maas-api-revoke-keys-e2e-ait-573837e3-8z48z\" (UID: \"d69cf7a3-856b-4cb8-aa7a-f978b93b14f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z"
Oct 30 00:54:31.098340 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:31.098321    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-573837e3-8z48z\" (UID: \"d69cf7a3-856b-4cb8-aa7a-f978b93b14f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z"
Oct 30 00:54:31.199578 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:31.199536    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-573837e3-8z48z\" (UID: \"d69cf7a3-856b-4cb8-aa7a-f978b93b14f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z"
Oct 30 00:54:31.199776 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:31.199600    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdhcr\" (UniqueName: \"kubernetes.io/projected/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-kube-api-access-tdhcr\") pod \"maas-api-revoke-keys-e2e-ait-573837e3-8z48z\" (UID: \"d69cf7a3-856b-4cb8-aa7a-f978b93b14f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z"
Oct 30 00:54:31.199853 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:31.199776    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-573837e3-8z48z\" (UID: \"d69cf7a3-856b-4cb8-aa7a-f978b93b14f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z"
Oct 30 00:54:31.200247 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:31.200227    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-573837e3-8z48z\" (UID: \"d69cf7a3-856b-4cb8-aa7a-f978b93b14f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z"
Oct 30 00:54:31.202237 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:31.202219    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-573837e3-8z48z\" (UID: \"d69cf7a3-856b-4cb8-aa7a-f978b93b14f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z"
Oct 30 00:54:31.222465 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:31.222444    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdhcr\" (UniqueName: \"kubernetes.io/projected/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-kube-api-access-tdhcr\") pod \"maas-api-revoke-keys-e2e-ait-573837e3-8z48z\" (UID: \"d69cf7a3-856b-4cb8-aa7a-f978b93b14f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z"
Oct 30 00:54:31.316784 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:31.316744    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z"
Oct 30 00:54:31.647203 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:54:31.647169    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69cf7a3_856b_4cb8_aa7a_f978b93b14f2.slice/crio-9ce100b22591e398dc8a1fd744cae155f334c09faef9b1e9cd13048a357053f9 WatchSource:0}: Error finding container 9ce100b22591e398dc8a1fd744cae155f334c09faef9b1e9cd13048a357053f9: Status 404 returned error can't find the container with id 9ce100b22591e398dc8a1fd744cae155f334c09faef9b1e9cd13048a357053f9
Oct 30 00:54:31.648719 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:31.648679    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z"]
Oct 30 00:54:32.000071 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:31.999992    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z" event={"ID":"d69cf7a3-856b-4cb8-aa7a-f978b93b14f2","Type":"ContainerStarted","Data":"9ce100b22591e398dc8a1fd744cae155f334c09faef9b1e9cd13048a357053f9"}
Oct 30 00:54:33.010413 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:33.010319    6099 generic.go:356] "Generic (PLEG): container finished" podID="d69cf7a3-856b-4cb8-aa7a-f978b93b14f2" containerID="f80244b1720ef86826d2356ae1d02200729cdcf82055d9abbd4b93957d4707ce" exitCode=0
Oct 30 00:54:33.010413 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:33.010401    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z" event={"ID":"d69cf7a3-856b-4cb8-aa7a-f978b93b14f2","Type":"ContainerDied","Data":"f80244b1720ef86826d2356ae1d02200729cdcf82055d9abbd4b93957d4707ce"}
Oct 30 00:54:34.228172 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:34.228139    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z"
Oct 30 00:54:34.383904 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:34.383872    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-kube-api-access-tdhcr\" (UniqueName: \"kubernetes.io/projected/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-kube-api-access-tdhcr\") pod \"d69cf7a3-856b-4cb8-aa7a-f978b93b14f2\" (UID: \"d69cf7a3-856b-4cb8-aa7a-f978b93b14f2\") "
Oct 30 00:54:34.384093 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:34.384077    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-maas-api-openshift-ca\") pod \"d69cf7a3-856b-4cb8-aa7a-f978b93b14f2\" (UID: \"d69cf7a3-856b-4cb8-aa7a-f978b93b14f2\") "
Oct 30 00:54:34.384142 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:34.384105    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-maas-api-xks-ca\") pod \"d69cf7a3-856b-4cb8-aa7a-f978b93b14f2\" (UID: \"d69cf7a3-856b-4cb8-aa7a-f978b93b14f2\") "
Oct 30 00:54:34.384418 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:34.384383    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-maas-api-openshift-ca" pod "d69cf7a3-856b-4cb8-aa7a-f978b93b14f2" (UID: "d69cf7a3-856b-4cb8-aa7a-f978b93b14f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:54:34.385801 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:34.385774    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:54:34.386748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:34.386290    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-kube-api-access-tdhcr" pod "d69cf7a3-856b-4cb8-aa7a-f978b93b14f2" (UID: "d69cf7a3-856b-4cb8-aa7a-f978b93b14f2"). InnerVolumeSpecName "kube-api-access-tdhcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:54:34.386864 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:34.386809    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-maas-api-xks-ca" pod "d69cf7a3-856b-4cb8-aa7a-f978b93b14f2" (UID: "d69cf7a3-856b-4cb8-aa7a-f978b93b14f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:54:34.486998 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:34.486964    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:54:34.486998 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:34.486990    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdhcr\" (UniqueName: \"kubernetes.io/projected/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2-kube-api-access-tdhcr\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:54:35.034074 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:35.034040    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z" event={"ID":"d69cf7a3-856b-4cb8-aa7a-f978b93b14f2","Type":"ContainerDied","Data":"9ce100b22591e398dc8a1fd744cae155f334c09faef9b1e9cd13048a357053f9"}
Oct 30 00:54:35.034226 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:35.034090    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ce100b22591e398dc8a1fd744cae155f334c09faef9b1e9cd13048a357053f9"
Oct 30 00:54:35.034226 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:35.034056    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z"
Oct 30 00:54:37.032539 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:37.032506    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw"]
Oct 30 00:54:37.044025 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:37.043990    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d283929f-ckrdw"]
Oct 30 00:54:38.263676 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:38.263640    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d85827-e8ab-4ca2-9e90-25dd27204143" path="/var/lib/kubelet/pods/46d85827-e8ab-4ca2-9e90-25dd27204143/volumes"
Oct 30 00:54:39.031824 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:39.031790    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj"]
Oct 30 00:54:39.043992 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:39.043963    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c4ad51-ctdqj"]
Oct 30 00:54:40.263867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:40.263835    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ece9619-1e8f-479c-8202-787f87779fdb" path="/var/lib/kubelet/pods/9ece9619-1e8f-479c-8202-787f87779fdb/volumes"
Oct 30 00:54:40.993775 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:40.993699    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6"
Oct 30 00:54:41.428019 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:41.427983    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g"]
Oct 30 00:54:45.711581 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:45.711538    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z"]
Oct 30 00:54:45.724462 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:45.724432    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-573837e3-8z48z"]
Oct 30 00:54:46.269225 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:46.269193    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d69cf7a3-856b-4cb8-aa7a-f978b93b14f2" path="/var/lib/kubelet/pods/d69cf7a3-856b-4cb8-aa7a-f978b93b14f2/volumes"
Oct 30 00:54:46.432863 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:46.432816    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g" podUID="8c07f884-b799-4e04-9e65-d59174993df5" containerName="payload-processing" containerID="cri-o://e7823e0f7d4a33625442ae267b6840eecb26faea0110a7d22f99d00f4ffba289" gracePeriod=25
Oct 30 00:54:46.633859 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:54:46.633818    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:54:46.782506 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:46.782033    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g"
Oct 30 00:54:46.931983 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:46.931895    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c07f884-b799-4e04-9e65-d59174993df5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c07f884-b799-4e04-9e65-d59174993df5-plugins-config-volume\") pod \"8c07f884-b799-4e04-9e65-d59174993df5\" (UID: \"8c07f884-b799-4e04-9e65-d59174993df5\") "
Oct 30 00:54:46.932159 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:46.932142    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c07f884-b799-4e04-9e65-d59174993df5-kube-api-access-2dkft\" (UniqueName: \"kubernetes.io/projected/8c07f884-b799-4e04-9e65-d59174993df5-kube-api-access-2dkft\") pod \"8c07f884-b799-4e04-9e65-d59174993df5\" (UID: \"8c07f884-b799-4e04-9e65-d59174993df5\") "
Oct 30 00:54:46.932314 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:46.932290    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c07f884-b799-4e04-9e65-d59174993df5-plugins-config-volume" pod "8c07f884-b799-4e04-9e65-d59174993df5" (UID: "8c07f884-b799-4e04-9e65-d59174993df5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:54:46.932784 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:46.932762    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c07f884-b799-4e04-9e65-d59174993df5-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:54:46.934374 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:46.934340    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c07f884-b799-4e04-9e65-d59174993df5-kube-api-access-2dkft" pod "8c07f884-b799-4e04-9e65-d59174993df5" (UID: "8c07f884-b799-4e04-9e65-d59174993df5"). InnerVolumeSpecName "kube-api-access-2dkft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:54:47.034671 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:47.034595    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dkft\" (UniqueName: \"kubernetes.io/projected/8c07f884-b799-4e04-9e65-d59174993df5-kube-api-access-2dkft\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:54:47.163563 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:47.163525    6099 generic.go:356] "Generic (PLEG): container finished" podID="8c07f884-b799-4e04-9e65-d59174993df5" containerID="e7823e0f7d4a33625442ae267b6840eecb26faea0110a7d22f99d00f4ffba289" exitCode=0
Oct 30 00:54:47.163771 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:47.163571    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g"
Oct 30 00:54:47.163771 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:47.163605    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g" event={"ID":"8c07f884-b799-4e04-9e65-d59174993df5","Type":"ContainerDied","Data":"e7823e0f7d4a33625442ae267b6840eecb26faea0110a7d22f99d00f4ffba289"}
Oct 30 00:54:47.163771 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:47.163679    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g" event={"ID":"8c07f884-b799-4e04-9e65-d59174993df5","Type":"ContainerDied","Data":"bd2b1243ed1091db24858ffa20df02170adc57c1493d4781f588e23f0940f2ba"}
Oct 30 00:54:47.163771 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:47.163701    6099 scope.go:122] "RemoveContainer" containerID="e7823e0f7d4a33625442ae267b6840eecb26faea0110a7d22f99d00f4ffba289"
Oct 30 00:54:47.181373 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:47.181353    6099 scope.go:122] "RemoveContainer" containerID="e7823e0f7d4a33625442ae267b6840eecb26faea0110a7d22f99d00f4ffba289"
Oct 30 00:54:47.181752 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:54:47.181689    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7823e0f7d4a33625442ae267b6840eecb26faea0110a7d22f99d00f4ffba289\": container with ID starting with e7823e0f7d4a33625442ae267b6840eecb26faea0110a7d22f99d00f4ffba289 not found: ID does not exist" containerID="e7823e0f7d4a33625442ae267b6840eecb26faea0110a7d22f99d00f4ffba289"
Oct 30 00:54:47.181820 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:47.181765    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7823e0f7d4a33625442ae267b6840eecb26faea0110a7d22f99d00f4ffba289"} err="failed to get container status \"e7823e0f7d4a33625442ae267b6840eecb26faea0110a7d22f99d00f4ffba289\": rpc error: code = NotFound desc = could not find container \"e7823e0f7d4a33625442ae267b6840eecb26faea0110a7d22f99d00f4ffba289\": container with ID starting with e7823e0f7d4a33625442ae267b6840eecb26faea0110a7d22f99d00f4ffba289 not found: ID does not exist"
Oct 30 00:54:47.217574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:47.217506    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g"]
Oct 30 00:54:47.229170 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:47.229144    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-573837e3-65cfb8bddd-mqv4g"]
Oct 30 00:54:48.109605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.109507    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"]
Oct 30 00:54:48.110670 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.110639    6099 cpu_manager.go:414] "containerMap: removing container" podUID="8c07f884-b799-4e04-9e65-d59174993df5" containerName="payload-processing"
Oct 30 00:54:48.110670 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.110669    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c07f884-b799-4e04-9e65-d59174993df5" containerName="payload-processing"
Oct 30 00:54:48.110958 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.110737    6099 cpu_manager.go:414] "containerMap: removing container" podUID="d69cf7a3-856b-4cb8-aa7a-f978b93b14f2" containerName="revoke-keys"
Oct 30 00:54:48.110958 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.110747    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d69cf7a3-856b-4cb8-aa7a-f978b93b14f2" containerName="revoke-keys"
Oct 30 00:54:48.111072 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.111031    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c07f884-b799-4e04-9e65-d59174993df5" containerName="payload-processing"
Oct 30 00:54:48.111072 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.111051    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d69cf7a3-856b-4cb8-aa7a-f978b93b14f2" containerName="revoke-keys"
Oct 30 00:54:48.115097 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.115074    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.140575 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.140540    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-bea3ef-openshift-default-dockercfg-hbbcs\""
Oct 30 00:54:48.190513 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.190459    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"]
Oct 30 00:54:48.278201 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.278128    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-podinfo\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.278469 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.278239    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-workload-socket\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.278469 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.278313    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-workload-certs\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.278469 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.278397    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-data\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.278469 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.278446    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-credential-socket\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.278837 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.278488    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-token\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.278837 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.278585    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrmvx\" (UniqueName: \"kubernetes.io/projected/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-kube-api-access-lrmvx\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.278837 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.278779    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-envoy\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.278837 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.278803    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istiod-ca-cert\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.302907 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.302870    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c07f884-b799-4e04-9e65-d59174993df5" path="/var/lib/kubelet/pods/8c07f884-b799-4e04-9e65-d59174993df5/volumes"
Oct 30 00:54:48.380446 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.380357    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-envoy\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.381077 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.381046    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istiod-ca-cert\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.381940 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.381919    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-podinfo\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.382569 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.382547    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-workload-socket\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.383453 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.383434    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-workload-certs\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.384010 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.383990    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-data\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.384416 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.384403    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-credential-socket\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.384736 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.384720    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-token\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.385137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.383015    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-workload-socket\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.385137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.384917    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmvx\" (UniqueName: \"kubernetes.io/projected/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-kube-api-access-lrmvx\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.385137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.384658    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-credential-socket\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.385137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.383863    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-workload-certs\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.385137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.381810    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istiod-ca-cert\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.385748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.384345    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-data\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.392578 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.392553    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-envoy\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.393355 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.393224    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-podinfo\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.409322 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.409281    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-token\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.409463 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.409371    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrmvx\" (UniqueName: \"kubernetes.io/projected/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-kube-api-access-lrmvx\") pod \"e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") " pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.435923 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.435895    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:48.844461 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.844414    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"]
Oct 30 00:54:48.849304 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:54:48.849279    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a3cb0b_00bf_4b7b_b4cb_976310413ee1.slice/crio-38eeae654e910ed2aabdea434f5935a3363e4e6a5019f11ab51110a9e185de42 WatchSource:0}: Error finding container 38eeae654e910ed2aabdea434f5935a3363e4e6a5019f11ab51110a9e185de42: Status 404 returned error can't find the container with id 38eeae654e910ed2aabdea434f5935a3363e4e6a5019f11ab51110a9e185de42
Oct 30 00:54:48.852625 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.852587    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:54:48.852840 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.852816    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:54:48.852988 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:48.852867    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:54:49.252564 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:49.252471    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n" event={"ID":"21a3cb0b-00bf-4b7b-b4cb-976310413ee1","Type":"ContainerStarted","Data":"5c286554dde69bbbf0c6e25c1cf1be7a3d46897f8996de161f1ad54216f76dd4"}
Oct 30 00:54:49.252564 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:49.252537    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n" event={"ID":"21a3cb0b-00bf-4b7b-b4cb-976310413ee1","Type":"ContainerStarted","Data":"38eeae654e910ed2aabdea434f5935a3363e4e6a5019f11ab51110a9e185de42"}
Oct 30 00:54:49.303580 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:49.302839    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n" podStartSLOduration=1.302635001 podStartE2EDuration="1.302635001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:49.298512648 +0000 UTC m=+1183.155420403" watchObservedRunningTime="2026-10-30 00:54:49.302635001 +0000 UTC m=+1183.159542756"
Oct 30 00:54:49.439284 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:49.436457    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:49.439499 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:49.439418    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 00:54:49.444770 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:49.443088    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n" podUID="21a3cb0b-00bf-4b7b-b4cb-976310413ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 00:54:50.033370 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:50.033335    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn"]
Oct 30 00:54:50.038041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:50.038017    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33fa5953-9r8hn"]
Oct 30 00:54:50.265930 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:50.265899    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e46913-e1f7-453a-a96a-ef57dee0a389" path="/var/lib/kubelet/pods/d1e46913-e1f7-453a-a96a-ef57dee0a389/volumes"
Oct 30 00:54:50.441117 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:50.441087    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:51.272620 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:51.272589    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:51.273791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:51.273771    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:54:54.038756 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:54.037020    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x"]
Oct 30 00:54:54.052734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:54.051696    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4"]
Oct 30 00:54:54.066119 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:54.066059    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11644b3-qlb6x"]
Oct 30 00:54:54.076660 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:54.076610    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f658383d-866h4"]
Oct 30 00:54:54.267906 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:54.267834    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09aabfe1-992e-40b3-b673-8f3d3af177a7" path="/var/lib/kubelet/pods/09aabfe1-992e-40b3-b673-8f3d3af177a7/volumes"
Oct 30 00:54:54.268615 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:54:54.268592    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5f5dc5-faba-4852-8639-5317f5cd1b7d" path="/var/lib/kubelet/pods/6a5f5dc5-faba-4852-8639-5317f5cd1b7d/volumes"
Oct 30 00:55:01.032429 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:01.032394    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94"]
Oct 30 00:55:01.043643 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:01.043615    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed772a4d-cnt94"]
Oct 30 00:55:02.263513 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:02.263477    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de33c43-4c4f-4293-9ab8-afe9ccd961d3" path="/var/lib/kubelet/pods/6de33c43-4c4f-4293-9ab8-afe9ccd961d3/volumes"
Oct 30 00:55:03.031758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:03.031725    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks"]
Oct 30 00:55:03.043445 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:03.043411    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a89488a-mpjks"]
Oct 30 00:55:04.264983 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:04.264953    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67e6c73a-6b09-46fe-8e70-93bbf68c41bf" path="/var/lib/kubelet/pods/67e6c73a-6b09-46fe-8e70-93bbf68c41bf/volumes"
Oct 30 00:55:06.480137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:06.480109    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 00:55:06.488036 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:06.488017    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 00:55:08.732837 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:08.732805    6099 scope.go:122] "RemoveContainer" containerID="044193e9e01b88b1fdc235767efa36bf30de04c4432aacd3bca57c80d24a40f9"
Oct 30 00:55:08.744096 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:08.744077    6099 scope.go:122] "RemoveContainer" containerID="2ad9b9848f85656b5068b046e6e9b2260d971dd7e683d9fbae6dc9f10b426968"
Oct 30 00:55:08.753967 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:08.753934    6099 scope.go:122] "RemoveContainer" containerID="49c4c00432a29b79b2a0a1a2a143079d1885f40978fd7eeb520d02862cf13d88"
Oct 30 00:55:08.764374 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:08.764353    6099 scope.go:122] "RemoveContainer" containerID="794035a630a453f002c1efb4d98cd06f10ae5a2625e39525bbb8b36af3d8bb39"
Oct 30 00:55:08.774065 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:08.774047    6099 scope.go:122] "RemoveContainer" containerID="f181a1988d8c69e50f32f70ea502a9759c72225447fc1ec54741ee85f9322912"
Oct 30 00:55:08.783685 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:08.783665    6099 scope.go:122] "RemoveContainer" containerID="e4bca2b37222c641cb18c3aa35c9d631acf406eb28647c9afa91f82c008ed5b3"
Oct 30 00:55:08.793477 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:08.793446    6099 scope.go:122] "RemoveContainer" containerID="dbec25116c6e6a37627e272a77dbf22b4cbb99dcb4cc25f30abbe2ba6bf1a283"
Oct 30 00:55:12.663928 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:12.663893    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg"]
Oct 30 00:55:12.670576 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:12.670556    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg"
Oct 30 00:55:12.704490 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:12.704459    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6867b8\""
Oct 30 00:55:12.712052 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:12.712024    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg"]
Oct 30 00:55:12.821732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:12.798762    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v477s\" (UniqueName: \"kubernetes.io/projected/0519dba7-7145-4761-b18c-397c0bee13d8-kube-api-access-v477s\") pod \"payload-processing-e2e-api-a-6867b8-757c78757b-7krfg\" (UID: \"0519dba7-7145-4761-b18c-397c0bee13d8\") " pod="openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg"
Oct 30 00:55:12.821732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:12.798864    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0519dba7-7145-4761-b18c-397c0bee13d8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6867b8-757c78757b-7krfg\" (UID: \"0519dba7-7145-4761-b18c-397c0bee13d8\") " pod="openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg"
Oct 30 00:55:12.901322 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:12.901270    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0519dba7-7145-4761-b18c-397c0bee13d8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6867b8-757c78757b-7krfg\" (UID: \"0519dba7-7145-4761-b18c-397c0bee13d8\") " pod="openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg"
Oct 30 00:55:12.901764 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:12.901736    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v477s\" (UniqueName: \"kubernetes.io/projected/0519dba7-7145-4761-b18c-397c0bee13d8-kube-api-access-v477s\") pod \"payload-processing-e2e-api-a-6867b8-757c78757b-7krfg\" (UID: \"0519dba7-7145-4761-b18c-397c0bee13d8\") " pod="openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg"
Oct 30 00:55:12.904477 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:12.904449    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0519dba7-7145-4761-b18c-397c0bee13d8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6867b8-757c78757b-7krfg\" (UID: \"0519dba7-7145-4761-b18c-397c0bee13d8\") " pod="openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg"
Oct 30 00:55:12.947771 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:12.944396    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v477s\" (UniqueName: \"kubernetes.io/projected/0519dba7-7145-4761-b18c-397c0bee13d8-kube-api-access-v477s\") pod \"payload-processing-e2e-api-a-6867b8-757c78757b-7krfg\" (UID: \"0519dba7-7145-4761-b18c-397c0bee13d8\") " pod="openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg"
Oct 30 00:55:12.995202 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:12.995173    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg"
Oct 30 00:55:13.433826 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:55:13.433515    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0519dba7_7145_4761_b18c_397c0bee13d8.slice/crio-80141954cca8197956e6dde324596cc5916c5e6dc26015b6af06d1f8c86d2563 WatchSource:0}: Error finding container 80141954cca8197956e6dde324596cc5916c5e6dc26015b6af06d1f8c86d2563: Status 404 returned error can't find the container with id 80141954cca8197956e6dde324596cc5916c5e6dc26015b6af06d1f8c86d2563
Oct 30 00:55:13.441165 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:13.441140    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg"]
Oct 30 00:55:13.496251 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:13.496157    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg" event={"ID":"0519dba7-7145-4761-b18c-397c0bee13d8","Type":"ContainerStarted","Data":"80141954cca8197956e6dde324596cc5916c5e6dc26015b6af06d1f8c86d2563"}
Oct 30 00:55:14.507248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:14.507213    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg" event={"ID":"0519dba7-7145-4761-b18c-397c0bee13d8","Type":"ContainerStarted","Data":"ec32240736c4e6473de34d4ef494e17744d70f751789fea6ed2d3c724a0dd2d2"}
Oct 30 00:55:14.507657 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:14.507277    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg"
Oct 30 00:55:14.552152 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:14.552094    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg" podStartSLOduration=2.552075834 podStartE2EDuration="2.552075834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:14.546947048 +0000 UTC m=+1208.403854802" watchObservedRunningTime="2026-10-30 00:55:14.552075834 +0000 UTC m=+1208.408983590"
Oct 30 00:55:16.033799 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:16.033763    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s"]
Oct 30 00:55:16.045014 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:16.044981    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-370faf19-g5t5s"]
Oct 30 00:55:16.282084 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:16.282049    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a997e50-2b16-4fab-8556-138de2d1cc11" path="/var/lib/kubelet/pods/1a997e50-2b16-4fab-8556-138de2d1cc11/volumes"
Oct 30 00:55:16.631122 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:55:16.631085    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:55:20.209729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:20.209293    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:55:25.518639 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:25.518609    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg"
Oct 30 00:55:28.031782 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:28.031749    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58"]
Oct 30 00:55:28.043308 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:28.043282    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-828c35b8-k9s58"]
Oct 30 00:55:28.263187 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:28.263153    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b652b50a-c0b1-423a-af82-5cc0357ec187" path="/var/lib/kubelet/pods/b652b50a-c0b1-423a-af82-5cc0357ec187/volumes"
Oct 30 00:55:29.505985 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.505951    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"]
Oct 30 00:55:29.513169 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.513133    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.534461 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.534433    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"]
Oct 30 00:55:29.539584 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.539560    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-353403-openshift-default-dockercfg-k8l4l\""
Oct 30 00:55:29.674733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.673879    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-credential-socket\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.674733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.673930    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-istio-envoy\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.674733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.674000    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a920d38-7878-4299-b399-64f3f76d7e55-istiod-ca-cert\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.674733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.674042    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-workload-socket\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.674733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.674082    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a920d38-7878-4299-b399-64f3f76d7e55-istio-token\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.674733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.674127    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-istio-data\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.674733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.674213    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-workload-certs\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.674733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.674235    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a920d38-7878-4299-b399-64f3f76d7e55-istio-podinfo\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.674733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.674462    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqmlk\" (UniqueName: \"kubernetes.io/projected/4a920d38-7878-4299-b399-64f3f76d7e55-kube-api-access-hqmlk\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.786466 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.786379    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a920d38-7878-4299-b399-64f3f76d7e55-istiod-ca-cert\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.787172 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.787138    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a920d38-7878-4299-b399-64f3f76d7e55-istiod-ca-cert\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.787559 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.787534    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-workload-socket\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.787850 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.787576    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-workload-socket\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.788059 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.788041    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a920d38-7878-4299-b399-64f3f76d7e55-istio-token\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.788233 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.788205    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-istio-data\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.788484 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.788455    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-workload-certs\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.788646 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.788627    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a920d38-7878-4299-b399-64f3f76d7e55-istio-podinfo\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.789023 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.788998    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-workload-certs\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.789175 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.789094    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqmlk\" (UniqueName: \"kubernetes.io/projected/4a920d38-7878-4299-b399-64f3f76d7e55-kube-api-access-hqmlk\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.789397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.789376    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-credential-socket\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.789511 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.789498    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-istio-envoy\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.789969 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.788696    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-istio-data\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.790347 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.790326    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-credential-socket\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.793745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.793693    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-istio-envoy\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.794406 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.794390    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a920d38-7878-4299-b399-64f3f76d7e55-istio-podinfo\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.812157 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.812126    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a920d38-7878-4299-b399-64f3f76d7e55-istio-token\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.814395 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.814371    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqmlk\" (UniqueName: \"kubernetes.io/projected/4a920d38-7878-4299-b399-64f3f76d7e55-kube-api-access-hqmlk\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-q56px\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:29.833080 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:29.832975    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:30.184325 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:55:30.184292    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a920d38_7878_4299_b399_64f3f76d7e55.slice/crio-7b1020bf9d944a58955b325bd336d13ca7eff37e9e0d358b058316696c1ee24b WatchSource:0}: Error finding container 7b1020bf9d944a58955b325bd336d13ca7eff37e9e0d358b058316696c1ee24b: Status 404 returned error can't find the container with id 7b1020bf9d944a58955b325bd336d13ca7eff37e9e0d358b058316696c1ee24b
Oct 30 00:55:30.184470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:30.184369    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"]
Oct 30 00:55:30.188456 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:30.188415    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:55:30.188569 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:30.188504    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:55:30.188569 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:30.188547    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:55:30.826556 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:30.826515    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px" event={"ID":"4a920d38-7878-4299-b399-64f3f76d7e55","Type":"ContainerStarted","Data":"89e1b21db3e8f12e1d52d09db4842f82174974bd6fd0c11bbd84bcffe7a11d75"}
Oct 30 00:55:30.827145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:30.826579    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px" event={"ID":"4a920d38-7878-4299-b399-64f3f76d7e55","Type":"ContainerStarted","Data":"7b1020bf9d944a58955b325bd336d13ca7eff37e9e0d358b058316696c1ee24b"}
Oct 30 00:55:30.833181 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:30.833161    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:30.887784 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:30.887727    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px" podStartSLOduration=1.8877002649999999 podStartE2EDuration="1.887700265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:30.88295216 +0000 UTC m=+1224.739859918" watchObservedRunningTime="2026-10-30 00:55:30.887700265 +0000 UTC m=+1224.744608110"
Oct 30 00:55:31.839125 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:31.839093    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:32.844656 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:32.844624    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:32.845773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:32.845749    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:55:35.741486 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:35.741447    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx"]
Oct 30 00:55:35.751820 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:35.751781    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx"
Oct 30 00:55:35.764938 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:35.764912    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx"]
Oct 30 00:55:35.770354 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:35.770331    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-353403\""
Oct 30 00:55:35.770469 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:35.770430    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-353403-dockercfg-nxwcr\""
Oct 30 00:55:35.805621 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:35.805350    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jff2j\" (UniqueName: \"kubernetes.io/projected/d6a88bf7-5dfd-4a6b-827a-4dbad7d58828-kube-api-access-jff2j\") pod \"payload-pre-processing-e2e-api-a-353403-75df964455-rttsx\" (UID: \"d6a88bf7-5dfd-4a6b-827a-4dbad7d58828\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx"
Oct 30 00:55:35.806286 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:35.806252    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a88bf7-5dfd-4a6b-827a-4dbad7d58828-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-353403-75df964455-rttsx\" (UID: \"d6a88bf7-5dfd-4a6b-827a-4dbad7d58828\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx"
Oct 30 00:55:35.962322 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:35.918664    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jff2j\" (UniqueName: \"kubernetes.io/projected/d6a88bf7-5dfd-4a6b-827a-4dbad7d58828-kube-api-access-jff2j\") pod \"payload-pre-processing-e2e-api-a-353403-75df964455-rttsx\" (UID: \"d6a88bf7-5dfd-4a6b-827a-4dbad7d58828\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx"
Oct 30 00:55:35.962322 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:35.924128    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a88bf7-5dfd-4a6b-827a-4dbad7d58828-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-353403-75df964455-rttsx\" (UID: \"d6a88bf7-5dfd-4a6b-827a-4dbad7d58828\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx"
Oct 30 00:55:35.962322 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:35.930937    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a88bf7-5dfd-4a6b-827a-4dbad7d58828-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-353403-75df964455-rttsx\" (UID: \"d6a88bf7-5dfd-4a6b-827a-4dbad7d58828\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx"
Oct 30 00:55:36.012791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:35.980402    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt"]
Oct 30 00:55:36.055481 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:36.042016    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt"]
Oct 30 00:55:36.055481 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:36.042145    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt"
Oct 30 00:55:36.055481 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:36.045764    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jff2j\" (UniqueName: \"kubernetes.io/projected/d6a88bf7-5dfd-4a6b-827a-4dbad7d58828-kube-api-access-jff2j\") pod \"payload-pre-processing-e2e-api-a-353403-75df964455-rttsx\" (UID: \"d6a88bf7-5dfd-4a6b-827a-4dbad7d58828\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx"
Oct 30 00:55:36.071822 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:36.070971    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx"
Oct 30 00:55:36.171008 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:36.170976    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqz2c\" (UniqueName: \"kubernetes.io/projected/984aa715-0b1d-450d-a28c-cc85e4beb848-kube-api-access-dqz2c\") pod \"payload-processing-e2e-api-a-353403-55845f89db-zx8nt\" (UID: \"984aa715-0b1d-450d-a28c-cc85e4beb848\") " pod="openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt"
Oct 30 00:55:36.171146 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:36.171104    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984aa715-0b1d-450d-a28c-cc85e4beb848-plugins-config-volume\") pod \"payload-processing-e2e-api-a-353403-55845f89db-zx8nt\" (UID: \"984aa715-0b1d-450d-a28c-cc85e4beb848\") " pod="openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt"
Oct 30 00:55:36.275806 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:36.273684    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqz2c\" (UniqueName: \"kubernetes.io/projected/984aa715-0b1d-450d-a28c-cc85e4beb848-kube-api-access-dqz2c\") pod \"payload-processing-e2e-api-a-353403-55845f89db-zx8nt\" (UID: \"984aa715-0b1d-450d-a28c-cc85e4beb848\") " pod="openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt"
Oct 30 00:55:36.275806 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:36.273842    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984aa715-0b1d-450d-a28c-cc85e4beb848-plugins-config-volume\") pod \"payload-processing-e2e-api-a-353403-55845f89db-zx8nt\" (UID: \"984aa715-0b1d-450d-a28c-cc85e4beb848\") " pod="openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt"
Oct 30 00:55:36.275806 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:36.274859    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984aa715-0b1d-450d-a28c-cc85e4beb848-plugins-config-volume\") pod \"payload-processing-e2e-api-a-353403-55845f89db-zx8nt\" (UID: \"984aa715-0b1d-450d-a28c-cc85e4beb848\") " pod="openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt"
Oct 30 00:55:36.298648 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:36.298016    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqz2c\" (UniqueName: \"kubernetes.io/projected/984aa715-0b1d-450d-a28c-cc85e4beb848-kube-api-access-dqz2c\") pod \"payload-processing-e2e-api-a-353403-55845f89db-zx8nt\" (UID: \"984aa715-0b1d-450d-a28c-cc85e4beb848\") " pod="openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt"
Oct 30 00:55:36.409306 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:36.402766    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt"
Oct 30 00:55:36.426166 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:36.410952    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx"]
Oct 30 00:55:36.822200 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:55:36.822161    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984aa715_0b1d_450d_a28c_cc85e4beb848.slice/crio-c4862e94d20f152ff02f6111f0b031cc597de02a55a353b2caf01cbff14c4e98 WatchSource:0}: Error finding container c4862e94d20f152ff02f6111f0b031cc597de02a55a353b2caf01cbff14c4e98: Status 404 returned error can't find the container with id c4862e94d20f152ff02f6111f0b031cc597de02a55a353b2caf01cbff14c4e98
Oct 30 00:55:36.825156 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:36.825128    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt"]
Oct 30 00:55:37.051599 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:37.051559    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt" event={"ID":"984aa715-0b1d-450d-a28c-cc85e4beb848","Type":"ContainerStarted","Data":"b5ea418f68adf5148147398c6ff891c14c4fbd0c1691cc658a9359bf42b43881"}
Oct 30 00:55:37.051798 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:37.051625    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt" event={"ID":"984aa715-0b1d-450d-a28c-cc85e4beb848","Type":"ContainerStarted","Data":"c4862e94d20f152ff02f6111f0b031cc597de02a55a353b2caf01cbff14c4e98"}
Oct 30 00:55:37.051798 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:37.051652    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt"
Oct 30 00:55:37.053546 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:37.053518    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx" event={"ID":"d6a88bf7-5dfd-4a6b-827a-4dbad7d58828","Type":"ContainerStarted","Data":"149e182606a19627d05ec99ce7af8a8987fd988d7cbec4916c21c92839b2d87f"}
Oct 30 00:55:37.053653 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:37.053565    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx" event={"ID":"d6a88bf7-5dfd-4a6b-827a-4dbad7d58828","Type":"ContainerStarted","Data":"56dd927a802271426b2f7418b5f5ef98a89991b023a173baef05aee793139726"}
Oct 30 00:55:37.053653 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:37.053606    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx"
Oct 30 00:55:37.096107 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:37.096012    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt" podStartSLOduration=2.095998565 podStartE2EDuration="2.095998565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:37.09298844 +0000 UTC m=+1230.949896197" watchObservedRunningTime="2026-10-30 00:55:37.095998565 +0000 UTC m=+1230.952906320"
Oct 30 00:55:37.135418 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:37.135366    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx" podStartSLOduration=2.135353381 podStartE2EDuration="2.135353381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:37.131761258 +0000 UTC m=+1230.988669009" watchObservedRunningTime="2026-10-30 00:55:37.135353381 +0000 UTC m=+1230.992261136"
Oct 30 00:55:42.035908 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:42.035869    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls"]
Oct 30 00:55:42.047728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:42.047676    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-800799d3-h6dls"]
Oct 30 00:55:42.265587 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:42.265552    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9fa6568-b070-4871-8188-6cec656f3094" path="/var/lib/kubelet/pods/d9fa6568-b070-4871-8188-6cec656f3094/volumes"
Oct 30 00:55:43.305994 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.305951    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal" podUID="f8c3a9edbd8cbf03175beab8c4db7242" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:43.722729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.722638    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"]
Oct 30 00:55:43.729373 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.729349    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.753405 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.753376    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"]
Oct 30 00:55:43.757316 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.757294    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-62f5d1-openshift-default-dockercfg-hprqt\""
Oct 30 00:55:43.823763 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.823726    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-envoy\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.823954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.823785    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-workload-certs\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.824009 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.823959    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-credential-socket\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.824082 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.824059    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-workload-socket\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.824157 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.824139    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-data\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.824667 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.824289    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-token\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.824667 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.824434    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-podinfo\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.824667 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.824525    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gjh7\" (UniqueName: \"kubernetes.io/projected/bdf12444-9b7c-4721-900b-44b11df44dd1-kube-api-access-7gjh7\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.824667 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.824563    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf12444-9b7c-4721-900b-44b11df44dd1-istiod-ca-cert\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.926698 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.926645    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-envoy\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.926944 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.926788    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-workload-certs\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.926944 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.926864    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-credential-socket\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.926944 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.926909    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-workload-socket\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.927091 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.926972    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-data\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.927091 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.927079    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-token\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.927224 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.927203    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-podinfo\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.927298 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.927281    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gjh7\" (UniqueName: \"kubernetes.io/projected/bdf12444-9b7c-4721-900b-44b11df44dd1-kube-api-access-7gjh7\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.927348 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.927323    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf12444-9b7c-4721-900b-44b11df44dd1-istiod-ca-cert\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.927906 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.927870    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-workload-socket\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.928195 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.928163    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-data\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.928322 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.928211    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf12444-9b7c-4721-900b-44b11df44dd1-istiod-ca-cert\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.928789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.928765    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-workload-certs\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.929047 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.929020    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-credential-socket\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.934839 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.934801    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-envoy\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.941697 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.941673    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-podinfo\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.952114 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.952090    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-token\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:43.952114 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:43.952097    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gjh7\" (UniqueName: \"kubernetes.io/projected/bdf12444-9b7c-4721-900b-44b11df44dd1-kube-api-access-7gjh7\") pod \"e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") " pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:44.044081 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:44.044037    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9"]
Oct 30 00:55:44.049092 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:44.049044    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3185456e-bwff9"]
Oct 30 00:55:44.049890 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:44.049858    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:44.287377 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:44.287343    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c877c1a-9a08-4cf6-a881-b8f19ca4433b" path="/var/lib/kubelet/pods/7c877c1a-9a08-4cf6-a881-b8f19ca4433b/volumes"
Oct 30 00:55:44.461771 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:44.460688    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"]
Oct 30 00:55:44.471204 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:55:44.471174    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf12444_9b7c_4721_900b_44b11df44dd1.slice/crio-f39897083432f617c459f65b49fd36917534fc53cc40cdcfea89a8a2e24596d9 WatchSource:0}: Error finding container f39897083432f617c459f65b49fd36917534fc53cc40cdcfea89a8a2e24596d9: Status 404 returned error can't find the container with id f39897083432f617c459f65b49fd36917534fc53cc40cdcfea89a8a2e24596d9
Oct 30 00:55:44.474657 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:44.474182    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:55:44.474657 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:44.474263    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:55:44.474657 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:44.474295    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:55:45.147666 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:45.146663    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72" event={"ID":"bdf12444-9b7c-4721-900b-44b11df44dd1","Type":"ContainerStarted","Data":"99574e2a6108acfee0ee05fcd0952a3b7d812f8531436b457ee9397de3e2a065"}
Oct 30 00:55:45.147666 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:45.146811    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72" event={"ID":"bdf12444-9b7c-4721-900b-44b11df44dd1","Type":"ContainerStarted","Data":"f39897083432f617c459f65b49fd36917534fc53cc40cdcfea89a8a2e24596d9"}
Oct 30 00:55:45.207417 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:45.207327    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72" podStartSLOduration=2.207305961 podStartE2EDuration="2.207305961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:45.206556244 +0000 UTC m=+1239.063463999" watchObservedRunningTime="2026-10-30 00:55:45.207305961 +0000 UTC m=+1239.064213721"
Oct 30 00:55:46.050070 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:46.050034    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:46.055265 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:46.055242    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:46.155622 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:46.155594    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:46.156650 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:46.156623    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:55:46.632255 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:55:46.632217    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:55:48.065613 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.065585    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt"
Oct 30 00:55:48.066024 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.065656    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx"
Oct 30 00:55:48.483084 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.483005    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"]
Oct 30 00:55:48.488542 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.488511    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.517908 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.517873    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"]
Oct 30 00:55:48.521374 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.520478    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-610065-openshift-default-dockercfg-lf28g\""
Oct 30 00:55:48.659524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.656555    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-workload-certs\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.659524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.656979    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9004633-73af-405e-8cd8-b1bd569602af-istiod-ca-cert\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.659524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.657096    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-credential-socket\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.659524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.657162    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-istio-data\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.659524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.657206    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9004633-73af-405e-8cd8-b1bd569602af-istio-podinfo\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.659524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.657266    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-workload-socket\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.659524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.657331    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9004633-73af-405e-8cd8-b1bd569602af-istio-token\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.659524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.657587    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8x6q\" (UniqueName: \"kubernetes.io/projected/a9004633-73af-405e-8cd8-b1bd569602af-kube-api-access-w8x6q\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.659524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.657668    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-istio-envoy\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.759153 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.759061    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-workload-certs\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.759336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.759206    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9004633-73af-405e-8cd8-b1bd569602af-istiod-ca-cert\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.759336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.759248    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-credential-socket\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.759336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.759283    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-istio-data\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.759336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.759321    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9004633-73af-405e-8cd8-b1bd569602af-istio-podinfo\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.759561 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.759372    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-workload-socket\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.759561 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.759430    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9004633-73af-405e-8cd8-b1bd569602af-istio-token\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.759561 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.759477    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8x6q\" (UniqueName: \"kubernetes.io/projected/a9004633-73af-405e-8cd8-b1bd569602af-kube-api-access-w8x6q\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.759767 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.759556    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-workload-certs\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.759767 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.759607    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-credential-socket\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.759767 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.759630    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-istio-envoy\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.759977 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.759960    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-istio-data\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.760088 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.760065    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-workload-socket\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.760148 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.760074    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9004633-73af-405e-8cd8-b1bd569602af-istiod-ca-cert\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.764555 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.763132    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9004633-73af-405e-8cd8-b1bd569602af-istio-podinfo\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.764872 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.764850    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-istio-envoy\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.788269 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.788230    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9004633-73af-405e-8cd8-b1bd569602af-istio-token\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.789869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.789826    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8x6q\" (UniqueName: \"kubernetes.io/projected/a9004633-73af-405e-8cd8-b1bd569602af-kube-api-access-w8x6q\") pod \"e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") " pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:48.811396 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:48.811353    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:49.181096 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:55:49.181051    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9004633_73af_405e_8cd8_b1bd569602af.slice/crio-f959cfd44ae2c96b412eaf41a1731da5d7fbdb3c618f4441a2db8fe42fef2fd1 WatchSource:0}: Error finding container f959cfd44ae2c96b412eaf41a1731da5d7fbdb3c618f4441a2db8fe42fef2fd1: Status 404 returned error can't find the container with id f959cfd44ae2c96b412eaf41a1731da5d7fbdb3c618f4441a2db8fe42fef2fd1
Oct 30 00:55:49.183033 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:49.183004    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"]
Oct 30 00:55:49.183862 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:49.183827    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:55:49.183931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:49.183900    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:55:49.183965 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:49.183932    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:55:50.204761 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:50.204718    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr" event={"ID":"a9004633-73af-405e-8cd8-b1bd569602af","Type":"ContainerStarted","Data":"c843d64a739e4001754f1931db315e4e062563754e08d3d9a1b14440f895b27d"}
Oct 30 00:55:50.205265 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:50.204783    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr" event={"ID":"a9004633-73af-405e-8cd8-b1bd569602af","Type":"ContainerStarted","Data":"f959cfd44ae2c96b412eaf41a1731da5d7fbdb3c618f4441a2db8fe42fef2fd1"}
Oct 30 00:55:50.278494 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:50.272189    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr" podStartSLOduration=2.272169398 podStartE2EDuration="2.272169398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:50.265187587 +0000 UTC m=+1244.122095343" watchObservedRunningTime="2026-10-30 00:55:50.272169398 +0000 UTC m=+1244.129077154"
Oct 30 00:55:50.813054 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:50.813018    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:50.818870 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:50.818840    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:51.215221 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:51.215139    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:51.216126 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:51.216104    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:55:53.147730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:53.146764    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b"]
Oct 30 00:55:53.197726 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:53.197529    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7149df7-jp96b"]
Oct 30 00:55:54.264969 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:54.264923    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e366629-56cd-4c9e-bb4f-e489885fae80" path="/var/lib/kubelet/pods/5e366629-56cd-4c9e-bb4f-e489885fae80/volumes"
Oct 30 00:55:58.148056 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:58.148017    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d"]
Oct 30 00:55:58.559236 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:55:58.559201    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f56e0e-vnc2d"]
Oct 30 00:56:00.266538 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:00.266500    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c42a5f-a350-4683-be19-4884131b1a42" path="/var/lib/kubelet/pods/d0c42a5f-a350-4683-be19-4884131b1a42/volumes"
Oct 30 00:56:02.032357 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:02.032323    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6"]
Oct 30 00:56:02.043758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:02.043725    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5720f7cd-vfqp6"]
Oct 30 00:56:02.265214 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:02.265168    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d71c5459-8203-4dbe-8cca-64cd23d7aefe" path="/var/lib/kubelet/pods/d71c5459-8203-4dbe-8cca-64cd23d7aefe/volumes"
Oct 30 00:56:04.697087 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:04.697046    6099 generic.go:356] "Generic (PLEG): container finished" podID="21a3cb0b-00bf-4b7b-b4cb-976310413ee1" containerID="5c286554dde69bbbf0c6e25c1cf1be7a3d46897f8996de161f1ad54216f76dd4" exitCode=0
Oct 30 00:56:04.697563 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:04.697112    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n" event={"ID":"21a3cb0b-00bf-4b7b-b4cb-976310413ee1","Type":"ContainerDied","Data":"5c286554dde69bbbf0c6e25c1cf1be7a3d46897f8996de161f1ad54216f76dd4"}
Oct 30 00:56:04.697743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:04.697684    6099 scope.go:122] "RemoveContainer" containerID="5c286554dde69bbbf0c6e25c1cf1be7a3d46897f8996de161f1ad54216f76dd4"
Oct 30 00:56:04.700182 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:04.700138    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:04.700290 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:04.700232    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:04.700290 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:04.700279    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:05.044787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.044387    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv"]
Oct 30 00:56:05.084729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.073299    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e68e111f-gzkgv"]
Oct 30 00:56:05.242923 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.242867    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"]
Oct 30 00:56:05.280517 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.280485    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"]
Oct 30 00:56:05.280902 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.280875    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.305159 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.305082    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a4fe0143-openshift-default-dockercfg-89vfc\""
Oct 30 00:56:05.307942 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.307915    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.308224 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.308206    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxbh6\" (UniqueName: \"kubernetes.io/projected/e764760a-aad7-44be-9068-387287c0602c-kube-api-access-rxbh6\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.308417 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.308402    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.309064 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.309035    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e764760a-aad7-44be-9068-387287c0602c-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.309959 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.309939    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.310216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.310200    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.310393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.310377    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e764760a-aad7-44be-9068-387287c0602c-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.310585 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.310567    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.310781 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.310762    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e764760a-aad7-44be-9068-387287c0602c-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.413001 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.412953    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbh6\" (UniqueName: \"kubernetes.io/projected/e764760a-aad7-44be-9068-387287c0602c-kube-api-access-rxbh6\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.413205 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.413050    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.413205 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.413151    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e764760a-aad7-44be-9068-387287c0602c-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.413336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.413305    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.413442 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.413418    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.413588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.413499    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e764760a-aad7-44be-9068-387287c0602c-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.413588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.413581    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.413806 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.413615    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e764760a-aad7-44be-9068-387287c0602c-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.413806 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.413698    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.414273 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.414238    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.420675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.416185    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.420675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.416770    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.420675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.417257    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.420675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.417918    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e764760a-aad7-44be-9068-387287c0602c-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.420675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.420079    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e764760a-aad7-44be-9068-387287c0602c-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.429366 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.429335    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.438339 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.438308    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxbh6\" (UniqueName: \"kubernetes.io/projected/e764760a-aad7-44be-9068-387287c0602c-kube-api-access-rxbh6\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.439300 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.439274    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e764760a-aad7-44be-9068-387287c0602c-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.612744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.612645    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:05.726939 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.726321    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n" event={"ID":"21a3cb0b-00bf-4b7b-b4cb-976310413ee1","Type":"ContainerStarted","Data":"b98ec03481824c514dc2f9a8cb0b26915c33aceba6e8a4395db92c787e2ef683"}
Oct 30 00:56:05.993659 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:56:05.993624    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode764760a_aad7_44be_9068_387287c0602c.slice/crio-b42959161a4b55da4bbeaa7913e4ce536211a5962cccd152877b2362c01cadfe WatchSource:0}: Error finding container b42959161a4b55da4bbeaa7913e4ce536211a5962cccd152877b2362c01cadfe: Status 404 returned error can't find the container with id b42959161a4b55da4bbeaa7913e4ce536211a5962cccd152877b2362c01cadfe
Oct 30 00:56:05.996365 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.996338    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"]
Oct 30 00:56:05.996954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.996922    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:05.997008 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.996987    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:05.997048 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:05.997015    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:06.308450 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:06.308413    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdc01056-d380-4980-b8db-2d2ed16d46fd" path="/var/lib/kubelet/pods/fdc01056-d380-4980-b8db-2d2ed16d46fd/volumes"
Oct 30 00:56:06.308837 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:06.308819    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:56:06.436525 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:06.436488    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:56:06.437981 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:06.437955    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 00:56:06.438114 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:06.438002    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n" podUID="21a3cb0b-00bf-4b7b-b4cb-976310413ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 00:56:06.737068 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:06.736968    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx" event={"ID":"e764760a-aad7-44be-9068-387287c0602c","Type":"ContainerStarted","Data":"4a3343d4a0da3b853c0b48dc5bb854a0fe56ba874dc6655339096fd7f6b7050a"}
Oct 30 00:56:06.737068 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:06.737032    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx" event={"ID":"e764760a-aad7-44be-9068-387287c0602c","Type":"ContainerStarted","Data":"b42959161a4b55da4bbeaa7913e4ce536211a5962cccd152877b2362c01cadfe"}
Oct 30 00:56:06.796329 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:06.795794    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx" podStartSLOduration=1.795699926 podStartE2EDuration="1.795699926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:06.788019467 +0000 UTC m=+1260.644927222" watchObservedRunningTime="2026-10-30 00:56:06.795699926 +0000 UTC m=+1260.652607685"
Oct 30 00:56:07.436830 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:07.436794    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 00:56:07.436991 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:07.436852    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n" podUID="21a3cb0b-00bf-4b7b-b4cb-976310413ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 00:56:07.644739 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:07.627727    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:07.684730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:07.671917    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:07.820728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:07.799295    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:07.820728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:07.814696    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:56:08.436418 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:08.436379    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 00:56:08.436590 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:08.436443    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n" podUID="21a3cb0b-00bf-4b7b-b4cb-976310413ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 00:56:08.892724 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:08.892683    6099 scope.go:122] "RemoveContainer" containerID="146bad53f9092afa1d86922a2e10956eed4e3ec2923c837b1f39c39a396f40d8"
Oct 30 00:56:08.905231 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:08.905207    6099 scope.go:122] "RemoveContainer" containerID="474a497954e2e8b8124542c734893f04f5f77abbed6daca4ba2756c9e12a9ea4"
Oct 30 00:56:08.920065 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:08.920045    6099 scope.go:122] "RemoveContainer" containerID="9575cd71200b8152996772a30a79107c79a9c2c9c2d5a2513e0c15d3bf049223"
Oct 30 00:56:08.930537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:08.930496    6099 scope.go:122] "RemoveContainer" containerID="9ce3278be7c6f8b318b0268176c7c5083ea609e6700d27826dc25277db9e5cec"
Oct 30 00:56:08.941202 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:08.941179    6099 scope.go:122] "RemoveContainer" containerID="d7826c4b4e8a2fdcc4d4eb84720b5b9e81d4a694f0df7ce905630deb756cb2bb"
Oct 30 00:56:08.951972 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:08.951951    6099 scope.go:122] "RemoveContainer" containerID="c9ed2c543ba8c76871f1695fd8162454bb23f95dfc638962f2d3d203bd11726d"
Oct 30 00:56:08.962107 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:08.962085    6099 scope.go:122] "RemoveContainer" containerID="2044a3dd2f5b8444d561b25d943ac0f8c251e4d67ca7358d858ba30b299994cc"
Oct 30 00:56:09.002148 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:09.002117    6099 scope.go:122] "RemoveContainer" containerID="2d9b4e149b15791982eff5a6e596f37fa0363eecd1d7b01172100e24473dc046"
Oct 30 00:56:09.013463 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:09.013442    6099 scope.go:122] "RemoveContainer" containerID="24b83bf894ee166f3e654d5fc6ebb3ef66e43b163f7cd510fb940d4dbe073861"
Oct 30 00:56:09.030972 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:09.030947    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5"]
Oct 30 00:56:09.043071 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:09.043043    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd5d58b3-79ch5"]
Oct 30 00:56:09.436775 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:09.436720    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 00:56:09.437045 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:09.436782    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n" podUID="21a3cb0b-00bf-4b7b-b4cb-976310413ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 00:56:10.287736 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:10.286780    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a02b4aa7-ea91-4b32-a463-0c1f9fca8e81" path="/var/lib/kubelet/pods/a02b4aa7-ea91-4b32-a463-0c1f9fca8e81/volumes"
Oct 30 00:56:10.436459 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:10.436385    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 00:56:10.436632 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:10.436493    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n" podUID="21a3cb0b-00bf-4b7b-b4cb-976310413ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 00:56:11.455728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:11.451948    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 00:56:11.455728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:11.452007    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n" podUID="21a3cb0b-00bf-4b7b-b4cb-976310413ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 00:56:12.441313 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.441273    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:56:12.442825 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.442797    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 00:56:12.582751 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.582093    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"]
Oct 30 00:56:12.589369 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.589341    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.604971 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.604924    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-73b32e-openshift-default-dockercfg-mknnz\""
Oct 30 00:56:12.613162 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.613131    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"]
Oct 30 00:56:12.760833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.739361    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e9d22c-b2b4-4795-be11-ff2d98383a92-istiod-ca-cert\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.760833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.739604    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-data\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.760833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.739726    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-podinfo\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.760833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.739793    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-credential-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.760833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.739831    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-workload-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.760833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.739862    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-workload-certs\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.760833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.739885    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-envoy\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.760833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.739912    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-token\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.760833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.739969    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfsm4\" (UniqueName: \"kubernetes.io/projected/19e9d22c-b2b4-4795-be11-ff2d98383a92-kube-api-access-vfsm4\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.856744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.851010    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-data\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.856744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.851143    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-podinfo\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.856744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.851217    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-credential-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.856744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.851258    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-workload-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.856744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.851292    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-workload-certs\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.856744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.851317    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-envoy\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.856744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.851365    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-token\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.856744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.851426    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfsm4\" (UniqueName: \"kubernetes.io/projected/19e9d22c-b2b4-4795-be11-ff2d98383a92-kube-api-access-vfsm4\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.856744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.851568    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e9d22c-b2b4-4795-be11-ff2d98383a92-istiod-ca-cert\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.856744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.852232    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-data\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.858026 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.857996    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-workload-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.858312 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.858292    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-credential-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.869535 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.859341    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e9d22c-b2b4-4795-be11-ff2d98383a92-istiod-ca-cert\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.869535 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.859570    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-workload-certs\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.917740 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.912222    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-envoy\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.921799 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.918959    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-podinfo\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.951680 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.951647    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfsm4\" (UniqueName: \"kubernetes.io/projected/19e9d22c-b2b4-4795-be11-ff2d98383a92-kube-api-access-vfsm4\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:12.961804 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:12.958588    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-token\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:13.212093 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:13.212056    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:13.593365 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:56:13.593319    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e9d22c_b2b4_4795_be11_ff2d98383a92.slice/crio-c3e728f58edd178b4e3c9c0d40c226b9db772799c90b0b478664f719802d0798 WatchSource:0}: Error finding container c3e728f58edd178b4e3c9c0d40c226b9db772799c90b0b478664f719802d0798: Status 404 returned error can't find the container with id c3e728f58edd178b4e3c9c0d40c226b9db772799c90b0b478664f719802d0798
Oct 30 00:56:13.595215 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:13.595187    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"]
Oct 30 00:56:13.596582 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:13.596546    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:13.596671 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:13.596614    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:13.596671 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:13.596647    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:13.908561 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:13.908474    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw" event={"ID":"19e9d22c-b2b4-4795-be11-ff2d98383a92","Type":"ContainerStarted","Data":"485ef597e39d43f5e309aec055ac6004c7ce23ed9b99c38bbb4b207671d7e6dc"}
Oct 30 00:56:13.908561 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:13.908541    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw" event={"ID":"19e9d22c-b2b4-4795-be11-ff2d98383a92","Type":"ContainerStarted","Data":"c3e728f58edd178b4e3c9c0d40c226b9db772799c90b0b478664f719802d0798"}
Oct 30 00:56:13.952726 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:13.952644    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw" podStartSLOduration=1.952623271 podStartE2EDuration="1.952623271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:13.951325723 +0000 UTC m=+1267.808233488" watchObservedRunningTime="2026-10-30 00:56:13.952623271 +0000 UTC m=+1267.809531027"
Oct 30 00:56:14.213031 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:14.212921    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:14.219480 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:14.219452    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:14.917778 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:14.917742    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:14.918823 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:14.918795    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:56:16.631036 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:56:16.630998    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:56:18.221316 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.221231    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq"]
Oct 30 00:56:18.225225 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.225202    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq"
Oct 30 00:56:18.245625 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.245599    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq"]
Oct 30 00:56:18.251519 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.251497    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-73b32e\""
Oct 30 00:56:18.251800 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.251776    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-73b32e-dockercfg-s6kkc\""
Oct 30 00:56:18.334731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.330245    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec09452-0400-491f-b1df-65d1e4160023-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq\" (UID: \"5ec09452-0400-491f-b1df-65d1e4160023\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq"
Oct 30 00:56:18.334731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.330348    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44h4t\" (UniqueName: \"kubernetes.io/projected/5ec09452-0400-491f-b1df-65d1e4160023-kube-api-access-44h4t\") pod \"payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq\" (UID: \"5ec09452-0400-491f-b1df-65d1e4160023\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq"
Oct 30 00:56:18.447727 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.445890    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec09452-0400-491f-b1df-65d1e4160023-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq\" (UID: \"5ec09452-0400-491f-b1df-65d1e4160023\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq"
Oct 30 00:56:18.447727 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.445971    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44h4t\" (UniqueName: \"kubernetes.io/projected/5ec09452-0400-491f-b1df-65d1e4160023-kube-api-access-44h4t\") pod \"payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq\" (UID: \"5ec09452-0400-491f-b1df-65d1e4160023\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq"
Oct 30 00:56:18.447727 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.447521    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec09452-0400-491f-b1df-65d1e4160023-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq\" (UID: \"5ec09452-0400-491f-b1df-65d1e4160023\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq"
Oct 30 00:56:18.590389 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.518494    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44h4t\" (UniqueName: \"kubernetes.io/projected/5ec09452-0400-491f-b1df-65d1e4160023-kube-api-access-44h4t\") pod \"payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq\" (UID: \"5ec09452-0400-491f-b1df-65d1e4160023\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq"
Oct 30 00:56:18.590389 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.544396    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq"
Oct 30 00:56:18.590389 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.565926    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv"]
Oct 30 00:56:18.686692 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.676074    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv"
Oct 30 00:56:18.706636 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.706591    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv"]
Oct 30 00:56:18.890142 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.890061    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42140e21-60da-460b-8d55-d7b9f5e53a2d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-73b32e-6655649548-fn9bv\" (UID: \"42140e21-60da-460b-8d55-d7b9f5e53a2d\") " pod="openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv"
Oct 30 00:56:18.890338 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.890325    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p7f6\" (UniqueName: \"kubernetes.io/projected/42140e21-60da-460b-8d55-d7b9f5e53a2d-kube-api-access-4p7f6\") pod \"payload-processing-e2e-api-b-73b32e-6655649548-fn9bv\" (UID: \"42140e21-60da-460b-8d55-d7b9f5e53a2d\") " pod="openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv"
Oct 30 00:56:18.993767 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.991900    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p7f6\" (UniqueName: \"kubernetes.io/projected/42140e21-60da-460b-8d55-d7b9f5e53a2d-kube-api-access-4p7f6\") pod \"payload-processing-e2e-api-b-73b32e-6655649548-fn9bv\" (UID: \"42140e21-60da-460b-8d55-d7b9f5e53a2d\") " pod="openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv"
Oct 30 00:56:18.993767 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.992214    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42140e21-60da-460b-8d55-d7b9f5e53a2d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-73b32e-6655649548-fn9bv\" (UID: \"42140e21-60da-460b-8d55-d7b9f5e53a2d\") " pod="openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv"
Oct 30 00:56:18.993767 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:18.993122    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42140e21-60da-460b-8d55-d7b9f5e53a2d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-73b32e-6655649548-fn9bv\" (UID: \"42140e21-60da-460b-8d55-d7b9f5e53a2d\") " pod="openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv"
Oct 30 00:56:19.015593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:19.015562    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p7f6\" (UniqueName: \"kubernetes.io/projected/42140e21-60da-460b-8d55-d7b9f5e53a2d-kube-api-access-4p7f6\") pod \"payload-processing-e2e-api-b-73b32e-6655649548-fn9bv\" (UID: \"42140e21-60da-460b-8d55-d7b9f5e53a2d\") " pod="openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv"
Oct 30 00:56:19.018178 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:56:19.018138    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec09452_0400_491f_b1df_65d1e4160023.slice/crio-b5fd101607e76326647849ab84d4097eaba9b27927259a99c73304e399c3c010 WatchSource:0}: Error finding container b5fd101607e76326647849ab84d4097eaba9b27927259a99c73304e399c3c010: Status 404 returned error can't find the container with id b5fd101607e76326647849ab84d4097eaba9b27927259a99c73304e399c3c010
Oct 30 00:56:19.019102 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:19.019077    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq"]
Oct 30 00:56:19.024802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:19.024781    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv"
Oct 30 00:56:19.387769 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:19.386076    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv"]
Oct 30 00:56:19.967020 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:19.966980    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq" event={"ID":"5ec09452-0400-491f-b1df-65d1e4160023","Type":"ContainerStarted","Data":"6797937017abb82841618acd3ea9f4d4c70e29f26ea77e8db87767c874402adc"}
Oct 30 00:56:19.967265 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:19.967047    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq" event={"ID":"5ec09452-0400-491f-b1df-65d1e4160023","Type":"ContainerStarted","Data":"b5fd101607e76326647849ab84d4097eaba9b27927259a99c73304e399c3c010"}
Oct 30 00:56:19.967265 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:19.967079    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq"
Oct 30 00:56:19.968750 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:19.968725    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv" event={"ID":"42140e21-60da-460b-8d55-d7b9f5e53a2d","Type":"ContainerStarted","Data":"a1f4f27955508b704b853f02dcc79108e1569d3441e9fa8a9fe917e920b73b73"}
Oct 30 00:56:19.968879 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:19.968775    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv"
Oct 30 00:56:19.968879 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:19.968799    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv" event={"ID":"42140e21-60da-460b-8d55-d7b9f5e53a2d","Type":"ContainerStarted","Data":"ed62f9f3a6c9582db7a8e5bad789bb3db123de283170bc283005605f864eae83"}
Oct 30 00:56:20.010201 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:20.010150    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq" podStartSLOduration=2.010135832 podStartE2EDuration="2.010135832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:20.006022452 +0000 UTC m=+1273.862930204" watchObservedRunningTime="2026-10-30 00:56:20.010135832 +0000 UTC m=+1273.867043624"
Oct 30 00:56:20.036419 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:20.036387    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv"]
Oct 30 00:56:20.041203 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:20.041177    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk"]
Oct 30 00:56:20.046335 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:20.046311    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e7f344-vj8wv"]
Oct 30 00:56:20.050422 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:20.050400    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b05b1b-bmvsk"]
Oct 30 00:56:20.052060 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:20.052012    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv" podStartSLOduration=2.051999185 podStartE2EDuration="2.051999185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:20.051023928 +0000 UTC m=+1273.907931683" watchObservedRunningTime="2026-10-30 00:56:20.051999185 +0000 UTC m=+1273.908906943"
Oct 30 00:56:20.265931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:20.265851    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df2972b5-c368-4a99-8ca1-7c0f0b271870" path="/var/lib/kubelet/pods/df2972b5-c368-4a99-8ca1-7c0f0b271870/volumes"
Oct 30 00:56:20.266522 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:20.266503    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e78b91bc-deda-4bc0-ac7d-c97e75bf6dce" path="/var/lib/kubelet/pods/e78b91bc-deda-4bc0-ac7d-c97e75bf6dce/volumes"
Oct 30 00:56:28.033183 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:28.033148    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4"]
Oct 30 00:56:28.044472 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:28.044445    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6"]
Oct 30 00:56:28.050140 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:28.050115    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa6f6-krmz4"]
Oct 30 00:56:28.055077 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:28.055054    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd583182-5g6n6"]
Oct 30 00:56:28.264427 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:28.264391    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb63c91-22fa-45a4-ac81-542c32cb95f7" path="/var/lib/kubelet/pods/9fb63c91-22fa-45a4-ac81-542c32cb95f7/volumes"
Oct 30 00:56:28.265008 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:28.264991    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8761662-0447-4cbc-bd62-a4d21a0b2ab5" path="/var/lib/kubelet/pods/a8761662-0447-4cbc-bd62-a4d21a0b2ab5/volumes"
Oct 30 00:56:30.980425 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:30.980394    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq"
Oct 30 00:56:30.980891 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:30.980463    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv"
Oct 30 00:56:34.034824 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:34.034788    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76"]
Oct 30 00:56:34.045060 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:34.045032    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dbd53e5-g5z76"]
Oct 30 00:56:34.264401 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:34.264366    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a97377e3-1b2f-4f3e-be46-b6511041b68c" path="/var/lib/kubelet/pods/a97377e3-1b2f-4f3e-be46-b6511041b68c/volumes"
Oct 30 00:56:34.598675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:34.598177    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:56:34.936531 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:34.936437    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"]
Oct 30 00:56:34.936781 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:34.936694    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml" podUID="c59323d4-d59d-421d-8c1e-00570828759a" containerName="istio-proxy" containerID="cri-o://8738655a94f271c11a6fa731543fafe2787d2f55dad36485cfedc11ef9204583" gracePeriod=30
Oct 30 00:56:34.950349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:34.950210    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"]
Oct 30 00:56:34.950921 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:34.950889    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7" podUID="ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" containerName="istio-proxy" containerID="cri-o://666d6027f515804e409c376da07601c3d50231167fd1fd38e07e0847473e91d0" gracePeriod=30
Oct 30 00:56:34.961418 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:34.961385    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"]
Oct 30 00:56:34.961657 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:34.961634    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv" podUID="14c6b739-42b1-4c75-8861-c051595f6620" containerName="istio-proxy" containerID="cri-o://a2477eb2294e8e625d92d9f8cb645b085fa4095a83d6e8037856496fad52279e" gracePeriod=30
Oct 30 00:56:36.041083 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:36.041040    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498"]
Oct 30 00:56:36.051081 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:36.051042    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ad393e-qt498"]
Oct 30 00:56:36.397796 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:36.393856    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f50d987e-97c1-4490-afd7-aa57f1b8561f" path="/var/lib/kubelet/pods/f50d987e-97c1-4490-afd7-aa57f1b8561f/volumes"
Oct 30 00:56:40.341839 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.341555    6099 generic.go:356] "Generic (PLEG): container finished" podID="c59323d4-d59d-421d-8c1e-00570828759a" containerID="8738655a94f271c11a6fa731543fafe2787d2f55dad36485cfedc11ef9204583" exitCode=0
Oct 30 00:56:40.341839 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.341777    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml" event={"ID":"c59323d4-d59d-421d-8c1e-00570828759a","Type":"ContainerDied","Data":"8738655a94f271c11a6fa731543fafe2787d2f55dad36485cfedc11ef9204583"}
Oct 30 00:56:40.344464 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.344334    6099 generic.go:356] "Generic (PLEG): container finished" podID="14c6b739-42b1-4c75-8861-c051595f6620" containerID="a2477eb2294e8e625d92d9f8cb645b085fa4095a83d6e8037856496fad52279e" exitCode=0
Oct 30 00:56:40.344464 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.344406    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv" event={"ID":"14c6b739-42b1-4c75-8861-c051595f6620","Type":"ContainerDied","Data":"a2477eb2294e8e625d92d9f8cb645b085fa4095a83d6e8037856496fad52279e"}
Oct 30 00:56:40.348381 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.348107    6099 generic.go:356] "Generic (PLEG): container finished" podID="ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" containerID="666d6027f515804e409c376da07601c3d50231167fd1fd38e07e0847473e91d0" exitCode=0
Oct 30 00:56:40.348381 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.348214    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7" event={"ID":"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad","Type":"ContainerDied","Data":"666d6027f515804e409c376da07601c3d50231167fd1fd38e07e0847473e91d0"}
Oct 30 00:56:40.445316 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.444895    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:56:40.546979 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.546950    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-credential-socket\") pod \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") "
Oct 30 00:56:40.547113 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.547091    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-podinfo\") pod \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") "
Oct 30 00:56:40.547738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.547423    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-credential-socket" pod "ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" (UID: "ad32816c-7a8e-47c7-9f1c-3dfea9f917ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:40.547912 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.547737    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-workload-certs\") pod \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") "
Oct 30 00:56:40.548031 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.547973    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-workload-certs" pod "ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" (UID: "ad32816c-7a8e-47c7-9f1c-3dfea9f917ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:40.548116 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.548100    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-token\" (UniqueName: \"kubernetes.io/projected/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-token\") pod \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") "
Oct 30 00:56:40.548375 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.548361    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istiod-ca-cert\") pod \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") "
Oct 30 00:56:40.548540 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.548527    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-workload-socket\") pod \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") "
Oct 30 00:56:40.548688 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.548677    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-envoy\") pod \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") "
Oct 30 00:56:40.548893 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.548876    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-data\") pod \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") "
Oct 30 00:56:40.549144 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.549124    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-kube-api-access-82fp6\" (UniqueName: \"kubernetes.io/projected/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-kube-api-access-82fp6\") pod \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\" (UID: \"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad\") "
Oct 30 00:56:40.549252 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.549232    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istiod-ca-cert" pod "ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" (UID: "ad32816c-7a8e-47c7-9f1c-3dfea9f917ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:56:40.549562 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.549365    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-data" pod "ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" (UID: "ad32816c-7a8e-47c7-9f1c-3dfea9f917ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:40.549825 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.549802    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-workload-socket" pod "ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" (UID: "ad32816c-7a8e-47c7-9f1c-3dfea9f917ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:40.550244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.550223    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:40.550391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.550246    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:40.550391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.550260    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:40.550391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.550272    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:40.550391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.550285    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:40.553307 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.553204    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-envoy" pod "ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" (UID: "ad32816c-7a8e-47c7-9f1c-3dfea9f917ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:40.553430 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.553403    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-token" pod "ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" (UID: "ad32816c-7a8e-47c7-9f1c-3dfea9f917ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:40.554136 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.554111    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-podinfo" pod "ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" (UID: "ad32816c-7a8e-47c7-9f1c-3dfea9f917ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:56:40.555627 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.555604    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-kube-api-access-82fp6" pod "ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" (UID: "ad32816c-7a8e-47c7-9f1c-3dfea9f917ad"). InnerVolumeSpecName "kube-api-access-82fp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:40.561788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.561769    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:56:40.652150 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.652115    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:40.652378 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.652359    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82fp6\" (UniqueName: \"kubernetes.io/projected/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-kube-api-access-82fp6\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:40.652492 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.652480    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:40.652579 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.652567    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:40.753476 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.753447    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14c6b739-42b1-4c75-8861-c051595f6620-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c6b739-42b1-4c75-8861-c051595f6620-istiod-ca-cert\") pod \"14c6b739-42b1-4c75-8861-c051595f6620\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") "
Oct 30 00:56:40.753600 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.753494    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-workload-certs\") pod \"14c6b739-42b1-4c75-8861-c051595f6620\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") "
Oct 30 00:56:40.753600 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.753534    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-istio-envoy\") pod \"14c6b739-42b1-4c75-8861-c051595f6620\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") "
Oct 30 00:56:40.753677 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.753607    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-workload-socket\") pod \"14c6b739-42b1-4c75-8861-c051595f6620\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") "
Oct 30 00:56:40.753832 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.753820    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14c6b739-42b1-4c75-8861-c051595f6620-istio-token\" (UniqueName: \"kubernetes.io/projected/14c6b739-42b1-4c75-8861-c051595f6620-istio-token\") pod \"14c6b739-42b1-4c75-8861-c051595f6620\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") "
Oct 30 00:56:40.753874 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.753845    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14c6b739-42b1-4c75-8861-c051595f6620-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c6b739-42b1-4c75-8861-c051595f6620-istio-podinfo\") pod \"14c6b739-42b1-4c75-8861-c051595f6620\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") "
Oct 30 00:56:40.753874 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.753870    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-istio-data\") pod \"14c6b739-42b1-4c75-8861-c051595f6620\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") "
Oct 30 00:56:40.753935 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.753912    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-credential-socket\") pod \"14c6b739-42b1-4c75-8861-c051595f6620\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") "
Oct 30 00:56:40.753971 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.753949    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14c6b739-42b1-4c75-8861-c051595f6620-kube-api-access-8pshx\" (UniqueName: \"kubernetes.io/projected/14c6b739-42b1-4c75-8861-c051595f6620-kube-api-access-8pshx\") pod \"14c6b739-42b1-4c75-8861-c051595f6620\" (UID: \"14c6b739-42b1-4c75-8861-c051595f6620\") "
Oct 30 00:56:40.754371 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.754231    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-workload-socket" pod "14c6b739-42b1-4c75-8861-c051595f6620" (UID: "14c6b739-42b1-4c75-8861-c051595f6620"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:40.755147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.754835    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14c6b739-42b1-4c75-8861-c051595f6620-istiod-ca-cert" pod "14c6b739-42b1-4c75-8861-c051595f6620" (UID: "14c6b739-42b1-4c75-8861-c051595f6620"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:56:40.755147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.754934    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:40.755466 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.755381    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-credential-socket" pod "14c6b739-42b1-4c75-8861-c051595f6620" (UID: "14c6b739-42b1-4c75-8861-c051595f6620"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:40.755466 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.755408    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-istio-data" pod "14c6b739-42b1-4c75-8861-c051595f6620" (UID: "14c6b739-42b1-4c75-8861-c051595f6620"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:40.755667 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.755646    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-workload-certs" pod "14c6b739-42b1-4c75-8861-c051595f6620" (UID: "14c6b739-42b1-4c75-8861-c051595f6620"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:40.757567 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.757520    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14c6b739-42b1-4c75-8861-c051595f6620-istio-podinfo" pod "14c6b739-42b1-4c75-8861-c051595f6620" (UID: "14c6b739-42b1-4c75-8861-c051595f6620"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:56:40.757693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.757562    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-istio-envoy" pod "14c6b739-42b1-4c75-8861-c051595f6620" (UID: "14c6b739-42b1-4c75-8861-c051595f6620"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:40.757693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.757550    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c6b739-42b1-4c75-8861-c051595f6620-istio-token" pod "14c6b739-42b1-4c75-8861-c051595f6620" (UID: "14c6b739-42b1-4c75-8861-c051595f6620"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:40.758303 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.758270    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c6b739-42b1-4c75-8861-c051595f6620-kube-api-access-8pshx" pod "14c6b739-42b1-4c75-8861-c051595f6620" (UID: "14c6b739-42b1-4c75-8861-c051595f6620"). InnerVolumeSpecName "kube-api-access-8pshx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:40.856802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.856692    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c6b739-42b1-4c75-8861-c051595f6620-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:40.856802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.856758    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c6b739-42b1-4c75-8861-c051595f6620-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:40.856802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.856772    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:40.856802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.856800    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:40.857129 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.856814    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pshx\" (UniqueName: \"kubernetes.io/projected/14c6b739-42b1-4c75-8861-c051595f6620-kube-api-access-8pshx\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:40.857129 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.856832    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c6b739-42b1-4c75-8861-c051595f6620-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:40.857129 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.856849    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:40.857129 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.856862    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c6b739-42b1-4c75-8861-c051595f6620-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:40.939571 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:40.939539    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:56:41.059220 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.059177    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-credential-socket\") pod \"c59323d4-d59d-421d-8c1e-00570828759a\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") "
Oct 30 00:56:41.059418 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.059317    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-workload-certs\") pod \"c59323d4-d59d-421d-8c1e-00570828759a\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") "
Oct 30 00:56:41.059418 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.059365    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c59323d4-d59d-421d-8c1e-00570828759a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59323d4-d59d-421d-8c1e-00570828759a-istiod-ca-cert\") pod \"c59323d4-d59d-421d-8c1e-00570828759a\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") "
Oct 30 00:56:41.059548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.059469    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c59323d4-d59d-421d-8c1e-00570828759a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59323d4-d59d-421d-8c1e-00570828759a-istio-podinfo\") pod \"c59323d4-d59d-421d-8c1e-00570828759a\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") "
Oct 30 00:56:41.059609 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.059542    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-workload-certs" pod "c59323d4-d59d-421d-8c1e-00570828759a" (UID: "c59323d4-d59d-421d-8c1e-00570828759a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:41.059676 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.059648    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-workload-socket\") pod \"c59323d4-d59d-421d-8c1e-00570828759a\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") "
Oct 30 00:56:41.059827 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.059812    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-istio-envoy\") pod \"c59323d4-d59d-421d-8c1e-00570828759a\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") "
Oct 30 00:56:41.059900 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.059836    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c59323d4-d59d-421d-8c1e-00570828759a-istio-token\" (UniqueName: \"kubernetes.io/projected/c59323d4-d59d-421d-8c1e-00570828759a-istio-token\") pod \"c59323d4-d59d-421d-8c1e-00570828759a\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") "
Oct 30 00:56:41.059900 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.059856    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c59323d4-d59d-421d-8c1e-00570828759a-kube-api-access-79q4v\" (UniqueName: \"kubernetes.io/projected/c59323d4-d59d-421d-8c1e-00570828759a-kube-api-access-79q4v\") pod \"c59323d4-d59d-421d-8c1e-00570828759a\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") "
Oct 30 00:56:41.060014 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.059972    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-istio-data\") pod \"c59323d4-d59d-421d-8c1e-00570828759a\" (UID: \"c59323d4-d59d-421d-8c1e-00570828759a\") "
Oct 30 00:56:41.060405 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.060384    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59323d4-d59d-421d-8c1e-00570828759a-istiod-ca-cert" pod "c59323d4-d59d-421d-8c1e-00570828759a" (UID: "c59323d4-d59d-421d-8c1e-00570828759a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:56:41.060589 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.060568    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-credential-socket" pod "c59323d4-d59d-421d-8c1e-00570828759a" (UID: "c59323d4-d59d-421d-8c1e-00570828759a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:41.060679 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.060628    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-workload-socket" pod "c59323d4-d59d-421d-8c1e-00570828759a" (UID: "c59323d4-d59d-421d-8c1e-00570828759a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:41.060679 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.060641    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:41.060679 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.060661    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59323d4-d59d-421d-8c1e-00570828759a-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:41.060899 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.060870    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-istio-data" pod "c59323d4-d59d-421d-8c1e-00570828759a" (UID: "c59323d4-d59d-421d-8c1e-00570828759a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:41.061877 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.061847    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c59323d4-d59d-421d-8c1e-00570828759a-istio-podinfo" pod "c59323d4-d59d-421d-8c1e-00570828759a" (UID: "c59323d4-d59d-421d-8c1e-00570828759a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:56:41.062577 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.062550    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59323d4-d59d-421d-8c1e-00570828759a-kube-api-access-79q4v" pod "c59323d4-d59d-421d-8c1e-00570828759a" (UID: "c59323d4-d59d-421d-8c1e-00570828759a"). InnerVolumeSpecName "kube-api-access-79q4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:41.063178 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.063151    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59323d4-d59d-421d-8c1e-00570828759a-istio-token" pod "c59323d4-d59d-421d-8c1e-00570828759a" (UID: "c59323d4-d59d-421d-8c1e-00570828759a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:41.063365 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.063341    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-istio-envoy" pod "c59323d4-d59d-421d-8c1e-00570828759a" (UID: "c59323d4-d59d-421d-8c1e-00570828759a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:41.162759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.162627    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:41.162759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.162662    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59323d4-d59d-421d-8c1e-00570828759a-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:41.162759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.162680    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79q4v\" (UniqueName: \"kubernetes.io/projected/c59323d4-d59d-421d-8c1e-00570828759a-kube-api-access-79q4v\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:41.162759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.162697    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:41.162759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.162739    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:41.162759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.162753    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59323d4-d59d-421d-8c1e-00570828759a-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:41.162759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.162767    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59323d4-d59d-421d-8c1e-00570828759a-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:41.361798 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.358866    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"
Oct 30 00:56:41.361798 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.358896    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv" event={"ID":"14c6b739-42b1-4c75-8861-c051595f6620","Type":"ContainerDied","Data":"458014f2c4348ef76f755bea74250e9a33fa2a68a6cf8f853a39c3d8187e633e"}
Oct 30 00:56:41.361798 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.359883    6099 scope.go:122] "RemoveContainer" containerID="a2477eb2294e8e625d92d9f8cb645b085fa4095a83d6e8037856496fad52279e"
Oct 30 00:56:41.361798 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.361491    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7" event={"ID":"ad32816c-7a8e-47c7-9f1c-3dfea9f917ad","Type":"ContainerDied","Data":"3dcc3dc507d06ade74bbc6a91f6747f224010ca9de8062e9704ae19fde03c1ed"}
Oct 30 00:56:41.361798 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.361569    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"
Oct 30 00:56:41.364466 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.364428    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"
Oct 30 00:56:41.364579 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.364460    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml" event={"ID":"c59323d4-d59d-421d-8c1e-00570828759a","Type":"ContainerDied","Data":"417ce9dc8bb695e075c927ae1dc3e3bd740ad9132f3768b673a0e3500a0d9f66"}
Oct 30 00:56:41.423329 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.422925    6099 scope.go:122] "RemoveContainer" containerID="666d6027f515804e409c376da07601c3d50231167fd1fd38e07e0847473e91d0"
Oct 30 00:56:41.436156 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.436076    6099 scope.go:122] "RemoveContainer" containerID="8738655a94f271c11a6fa731543fafe2787d2f55dad36485cfedc11ef9204583"
Oct 30 00:56:41.538514 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.536833    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"]
Oct 30 00:56:41.555065 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.554902    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-4zpb7"]
Oct 30 00:56:41.560163 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.560128    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"]
Oct 30 00:56:41.578250 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.578217    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-9hgfv"]
Oct 30 00:56:41.637715 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.637574    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"]
Oct 30 00:56:41.644484 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:41.644374    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-m7jml"]
Oct 30 00:56:42.303270 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:42.303236    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14c6b739-42b1-4c75-8861-c051595f6620" path="/var/lib/kubelet/pods/14c6b739-42b1-4c75-8861-c051595f6620/volumes"
Oct 30 00:56:42.303782 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:42.303767    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" path="/var/lib/kubelet/pods/ad32816c-7a8e-47c7-9f1c-3dfea9f917ad/volumes"
Oct 30 00:56:42.304233 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:42.304220    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c59323d4-d59d-421d-8c1e-00570828759a" path="/var/lib/kubelet/pods/c59323d4-d59d-421d-8c1e-00570828759a/volumes"
Oct 30 00:56:44.759160 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.759122    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"]
Oct 30 00:56:44.760228 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.760208    6099 cpu_manager.go:414] "containerMap: removing container" podUID="ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" containerName="istio-proxy"
Oct 30 00:56:44.760299 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.760232    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" containerName="istio-proxy"
Oct 30 00:56:44.760299 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.760282    6099 cpu_manager.go:414] "containerMap: removing container" podUID="14c6b739-42b1-4c75-8861-c051595f6620" containerName="istio-proxy"
Oct 30 00:56:44.760299 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.760288    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14c6b739-42b1-4c75-8861-c051595f6620" containerName="istio-proxy"
Oct 30 00:56:44.760413 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.760307    6099 cpu_manager.go:414] "containerMap: removing container" podUID="c59323d4-d59d-421d-8c1e-00570828759a" containerName="istio-proxy"
Oct 30 00:56:44.760413 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.760312    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59323d4-d59d-421d-8c1e-00570828759a" containerName="istio-proxy"
Oct 30 00:56:44.760480 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.760457    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="14c6b739-42b1-4c75-8861-c051595f6620" containerName="istio-proxy"
Oct 30 00:56:44.760480 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.760473    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59323d4-d59d-421d-8c1e-00570828759a" containerName="istio-proxy"
Oct 30 00:56:44.760582 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.760489    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad32816c-7a8e-47c7-9f1c-3dfea9f917ad" containerName="istio-proxy"
Oct 30 00:56:44.860454 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.856767    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"]
Oct 30 00:56:44.862139 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.861833    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:44.977171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.975681    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-envoy\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:44.977171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.975838    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-workload-socket\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:44.977171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.975941    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfgwx\" (UniqueName: \"kubernetes.io/projected/e82201b7-f285-4ec3-ba2c-ddb29d770391-kube-api-access-nfgwx\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:44.977171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.976001    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-podinfo\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:44.977171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.976049    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-workload-certs\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:44.977171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.976179    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-token\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:44.977171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.976261    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e82201b7-f285-4ec3-ba2c-ddb29d770391-istiod-ca-cert\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:44.977171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.976349    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-data\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:44.977171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:44.976382    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-credential-socket\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.087456 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.087425    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-podinfo\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.087669 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.087473    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-workload-certs\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.087669 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.087561    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-token\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.087669 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.087620    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e82201b7-f285-4ec3-ba2c-ddb29d770391-istiod-ca-cert\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.087869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.087687    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-data\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.087869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.087735    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-credential-socket\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.087869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.087830    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-envoy\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.088034 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.087915    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-workload-socket\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.088034 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.088012    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfgwx\" (UniqueName: \"kubernetes.io/projected/e82201b7-f285-4ec3-ba2c-ddb29d770391-kube-api-access-nfgwx\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.088142 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.088024    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-workload-certs\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.088142 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.088105    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-data\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.088506 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.088479    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e82201b7-f285-4ec3-ba2c-ddb29d770391-istiod-ca-cert\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.089014 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.088978    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-credential-socket\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.089014 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.089000    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-workload-socket\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.090049 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.090023    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-envoy\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.090172 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.090152    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-podinfo\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.110649 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.110611    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfgwx\" (UniqueName: \"kubernetes.io/projected/e82201b7-f285-4ec3-ba2c-ddb29d770391-kube-api-access-nfgwx\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.111111 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.111080    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-token\") pod \"e2e-api-a-353403-openshift-default-5f567587b9-sfvmj\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") " pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.282957 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.282920    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:45.880101 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.880073    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"]
Oct 30 00:56:45.880810 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:56:45.880785    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82201b7_f285_4ec3_ba2c_ddb29d770391.slice/crio-4b8e717a71e4f855276c021a87b225a8baa1b1534b3549b871a9ced2b1413045 WatchSource:0}: Error finding container 4b8e717a71e4f855276c021a87b225a8baa1b1534b3549b871a9ced2b1413045: Status 404 returned error can't find the container with id 4b8e717a71e4f855276c021a87b225a8baa1b1534b3549b871a9ced2b1413045
Oct 30 00:56:45.884455 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.884420    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:45.884540 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.884487    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:45.884540 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.884518    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:45.971349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:45.971319    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"]
Oct 30 00:56:46.032923 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.032884    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"]
Oct 30 00:56:46.033110 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.032997    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.051004 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.050976    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a32412-openshift-default-dockercfg-z6mfj\""
Oct 30 00:56:46.301786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.295321    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a75432-1847-4c51-96e8-ec245d55a103-istiod-ca-cert\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.301786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.295529    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-istio-envoy\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.301786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.295656    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-istio-data\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.301786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.295775    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-workload-certs\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.301786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.295879    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmcxz\" (UniqueName: \"kubernetes.io/projected/29a75432-1847-4c51-96e8-ec245d55a103-kube-api-access-mmcxz\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.301786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.295968    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-credential-socket\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.301786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.296048    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a75432-1847-4c51-96e8-ec245d55a103-istio-podinfo\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.301786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.296112    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-workload-socket\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.301786 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.296235    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a75432-1847-4c51-96e8-ec245d55a103-istio-token\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.400562 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.400496    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-istio-envoy\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.402278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.402250    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-istio-data\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.408308 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.408231    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-workload-certs\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.408568 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.408542    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmcxz\" (UniqueName: \"kubernetes.io/projected/29a75432-1847-4c51-96e8-ec245d55a103-kube-api-access-mmcxz\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.408788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.408760    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-credential-socket\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.408981 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.408963    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a75432-1847-4c51-96e8-ec245d55a103-istio-podinfo\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.409244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.409226    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-workload-socket\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.409505 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.409489    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a75432-1847-4c51-96e8-ec245d55a103-istio-token\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.409983 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.409962    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a75432-1847-4c51-96e8-ec245d55a103-istiod-ca-cert\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.411309 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.411151    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a75432-1847-4c51-96e8-ec245d55a103-istiod-ca-cert\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.453975 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.426616    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-istio-envoy\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.454206 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.427697    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-istio-data\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.454307 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.428182    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-workload-socket\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.454396 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.428639    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-workload-certs\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.454506 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.454417    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-credential-socket\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.485335 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.485257    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmcxz\" (UniqueName: \"kubernetes.io/projected/29a75432-1847-4c51-96e8-ec245d55a103-kube-api-access-mmcxz\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.490731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.490383    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a75432-1847-4c51-96e8-ec245d55a103-istio-podinfo\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.490731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.490466    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a75432-1847-4c51-96e8-ec245d55a103-istio-token\") pod \"e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") " pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:46.512256 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.512220    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj" event={"ID":"e82201b7-f285-4ec3-ba2c-ddb29d770391","Type":"ContainerStarted","Data":"11eb2ba27d3e81ac698a5c7f261a5db3a0f6b163ca92c8f94a03719edb61dba0"}
Oct 30 00:56:46.512435 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.512274    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj" event={"ID":"e82201b7-f285-4ec3-ba2c-ddb29d770391","Type":"ContainerStarted","Data":"4b8e717a71e4f855276c021a87b225a8baa1b1534b3549b871a9ced2b1413045"}
Oct 30 00:56:46.561357 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.561235    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj" podStartSLOduration=2.56121105 podStartE2EDuration="2.56121105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:46.556849101 +0000 UTC m=+1300.413756858" watchObservedRunningTime="2026-10-30 00:56:46.56121105 +0000 UTC m=+1300.418118826"
Oct 30 00:56:46.632355 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:56:46.632306    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:56:46.737767 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:46.736900    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:47.124680 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:47.124607    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"]
Oct 30 00:56:47.128648 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:56:47.128604    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a75432_1847_4c51_96e8_ec245d55a103.slice/crio-61732f2435524e66db469f5bbe85ea8504674e0d981af19ad346c6fc9f903e38 WatchSource:0}: Error finding container 61732f2435524e66db469f5bbe85ea8504674e0d981af19ad346c6fc9f903e38: Status 404 returned error can't find the container with id 61732f2435524e66db469f5bbe85ea8504674e0d981af19ad346c6fc9f903e38
Oct 30 00:56:47.132113 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:47.132071    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:47.132237 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:47.132170    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:47.132237 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:47.132202    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:47.271055 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:47.270644    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:56:47.271055 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:47.270732    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72" podUID="bdf12444-9b7c-4721-900b-44b11df44dd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:47.283990 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:47.283880    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:47.287632 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:47.287516    6099 patch_prober.go:29] interesting pod/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 00:56:47.287632 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:47.287561    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj" podUID="e82201b7-f285-4ec3-ba2c-ddb29d770391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 00:56:47.529817 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:47.529741    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn" event={"ID":"29a75432-1847-4c51-96e8-ec245d55a103","Type":"ContainerStarted","Data":"784d620e38d6a18ac1b3cc95e1db0dada1e1913cc211c848857531f968a98119"}
Oct 30 00:56:47.530151 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:47.530125    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn" event={"ID":"29a75432-1847-4c51-96e8-ec245d55a103","Type":"ContainerStarted","Data":"61732f2435524e66db469f5bbe85ea8504674e0d981af19ad346c6fc9f903e38"}
Oct 30 00:56:47.600048 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:47.599972    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn" podStartSLOduration=2.599948309 podStartE2EDuration="2.599948309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:47.578238024 +0000 UTC m=+1301.435145779" watchObservedRunningTime="2026-10-30 00:56:47.599948309 +0000 UTC m=+1301.456856056"
Oct 30 00:56:47.746886 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:47.746849    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:48.284734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:48.284370    6099 patch_prober.go:29] interesting pod/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 00:56:48.284734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:48.284428    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj" podUID="e82201b7-f285-4ec3-ba2c-ddb29d770391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 00:56:48.744377 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:48.744343    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:48.894808 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:48.891816    6099 patch_prober.go:29] interesting pod/e2e-api-a-353403-openshift-default-5f567587b9-q56px container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:56:48.894808 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:48.891883    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px" podUID="4a920d38-7878-4299-b399-64f3f76d7e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:49.139127 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.139089    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4"]
Oct 30 00:56:49.157401 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.157371    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4"
Oct 30 00:56:49.163886 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.163853    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4"]
Oct 30 00:56:49.179615 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.178503    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:56:49.179615 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.179221    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:56:49.179615 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.179281    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:56:49.179615 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.179380    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:56:49.238359 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.238319    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c744876-e4fc-4433-8bcd-7c490336f1da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8090513-txdb4\" (UID: \"7c744876-e4fc-4433-8bcd-7c490336f1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4"
Oct 30 00:56:49.238553 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.238502    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c744876-e4fc-4433-8bcd-7c490336f1da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8090513-txdb4\" (UID: \"7c744876-e4fc-4433-8bcd-7c490336f1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4"
Oct 30 00:56:49.238631 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.238611    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksv4s\" (UniqueName: \"kubernetes.io/projected/7c744876-e4fc-4433-8bcd-7c490336f1da-kube-api-access-ksv4s\") pod \"maas-api-revoke-keys-e2e-mt-d8090513-txdb4\" (UID: \"7c744876-e4fc-4433-8bcd-7c490336f1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4"
Oct 30 00:56:49.301732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.286351    6099 patch_prober.go:29] interesting pod/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 00:56:49.301732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.286413    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj" podUID="e82201b7-f285-4ec3-ba2c-ddb29d770391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 00:56:49.340440 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.340193    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c744876-e4fc-4433-8bcd-7c490336f1da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8090513-txdb4\" (UID: \"7c744876-e4fc-4433-8bcd-7c490336f1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4"
Oct 30 00:56:49.340440 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.340418    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c744876-e4fc-4433-8bcd-7c490336f1da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8090513-txdb4\" (UID: \"7c744876-e4fc-4433-8bcd-7c490336f1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4"
Oct 30 00:56:49.340831 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.340536    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksv4s\" (UniqueName: \"kubernetes.io/projected/7c744876-e4fc-4433-8bcd-7c490336f1da-kube-api-access-ksv4s\") pod \"maas-api-revoke-keys-e2e-mt-d8090513-txdb4\" (UID: \"7c744876-e4fc-4433-8bcd-7c490336f1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4"
Oct 30 00:56:49.341761 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.341093    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c744876-e4fc-4433-8bcd-7c490336f1da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8090513-txdb4\" (UID: \"7c744876-e4fc-4433-8bcd-7c490336f1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4"
Oct 30 00:56:49.346964 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.346906    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c744876-e4fc-4433-8bcd-7c490336f1da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8090513-txdb4\" (UID: \"7c744876-e4fc-4433-8bcd-7c490336f1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4"
Oct 30 00:56:49.414572 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.414489    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksv4s\" (UniqueName: \"kubernetes.io/projected/7c744876-e4fc-4433-8bcd-7c490336f1da-kube-api-access-ksv4s\") pod \"maas-api-revoke-keys-e2e-mt-d8090513-txdb4\" (UID: \"7c744876-e4fc-4433-8bcd-7c490336f1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4"
Oct 30 00:56:49.549471 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.549438    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:49.550519 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.550497    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:56:49.563903 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:49.563879    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4"
Oct 30 00:56:50.356081 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:56:50.106122    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c744876_e4fc_4433_8bcd_7c490336f1da.slice/crio-baf95a506a3e15add2a057e13bd72bb0b555fea76fd936b084852b6986282c57 WatchSource:0}: Error finding container baf95a506a3e15add2a057e13bd72bb0b555fea76fd936b084852b6986282c57: Status 404 returned error can't find the container with id baf95a506a3e15add2a057e13bd72bb0b555fea76fd936b084852b6986282c57
Oct 30 00:56:50.356081 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:50.106671    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4"]
Oct 30 00:56:50.356081 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:50.297057    6099 patch_prober.go:29] interesting pod/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 00:56:50.356081 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:50.297117    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj" podUID="e82201b7-f285-4ec3-ba2c-ddb29d770391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 00:56:50.561241 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:50.561206    6099 generic.go:356] "Generic (PLEG): container finished" podID="4a920d38-7878-4299-b399-64f3f76d7e55" containerID="89e1b21db3e8f12e1d52d09db4842f82174974bd6fd0c11bbd84bcffe7a11d75" exitCode=0
Oct 30 00:56:50.561452 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:50.561289    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px" event={"ID":"4a920d38-7878-4299-b399-64f3f76d7e55","Type":"ContainerDied","Data":"89e1b21db3e8f12e1d52d09db4842f82174974bd6fd0c11bbd84bcffe7a11d75"}
Oct 30 00:56:50.561823 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:50.561800    6099 scope.go:122] "RemoveContainer" containerID="89e1b21db3e8f12e1d52d09db4842f82174974bd6fd0c11bbd84bcffe7a11d75"
Oct 30 00:56:50.562818 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:50.562792    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4" event={"ID":"7c744876-e4fc-4433-8bcd-7c490336f1da","Type":"ContainerStarted","Data":"baf95a506a3e15add2a057e13bd72bb0b555fea76fd936b084852b6986282c57"}
Oct 30 00:56:50.563994 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:50.563964    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:50.564065 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:50.564032    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:50.564103 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:50.564063    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:51.291785 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:51.289449    6099 patch_prober.go:29] interesting pod/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 00:56:51.291785 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:51.289517    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj" podUID="e82201b7-f285-4ec3-ba2c-ddb29d770391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 00:56:51.597007 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:51.583126    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px" event={"ID":"4a920d38-7878-4299-b399-64f3f76d7e55","Type":"ContainerStarted","Data":"9e0a6e0513f77dadd687fcfba21a0822533c4e25c145b08ae501f2fde03d66d0"}
Oct 30 00:56:51.597007 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:51.588087    6099 generic.go:356] "Generic (PLEG): container finished" podID="7c744876-e4fc-4433-8bcd-7c490336f1da" containerID="99be46bd352a142f44fe3095772b251e059bfd23a894060cbe940931c8818d13" exitCode=0
Oct 30 00:56:51.597007 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:51.588183    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4" event={"ID":"7c744876-e4fc-4433-8bcd-7c490336f1da","Type":"ContainerDied","Data":"99be46bd352a142f44fe3095772b251e059bfd23a894060cbe940931c8818d13"}
Oct 30 00:56:51.597007 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:51.590486    6099 generic.go:356] "Generic (PLEG): container finished" podID="bdf12444-9b7c-4721-900b-44b11df44dd1" containerID="99574e2a6108acfee0ee05fcd0952a3b7d812f8531436b457ee9397de3e2a065" exitCode=0
Oct 30 00:56:51.597007 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:51.591722    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72" event={"ID":"bdf12444-9b7c-4721-900b-44b11df44dd1","Type":"ContainerDied","Data":"99574e2a6108acfee0ee05fcd0952a3b7d812f8531436b457ee9397de3e2a065"}
Oct 30 00:56:51.597007 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:51.592169    6099 scope.go:122] "RemoveContainer" containerID="99574e2a6108acfee0ee05fcd0952a3b7d812f8531436b457ee9397de3e2a065"
Oct 30 00:56:51.597007 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:51.594575    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:51.597007 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:51.594666    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:51.597007 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:51.594732    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:51.838859 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:51.838127    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:56:51.838859 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:51.838249    6099 patch_prober.go:29] interesting pod/e2e-api-a-353403-openshift-default-5f567587b9-q56px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 00:56:51.838859 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:51.838292    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px" podUID="4a920d38-7878-4299-b399-64f3f76d7e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 00:56:52.043745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.042911    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4"]
Oct 30 00:56:52.049739 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.049694    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ff1f149-zv6t4"]
Oct 30 00:56:52.062036 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.062007    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4"]
Oct 30 00:56:52.082995 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.082950    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4"
Oct 30 00:56:52.128024 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.118430    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a32412-dockercfg-xdqm6\""
Oct 30 00:56:52.287738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.286803    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h45xr\" (UniqueName: \"kubernetes.io/projected/0e190a98-17fe-46f6-adcf-34223eaf820e-kube-api-access-h45xr\") pod \"payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4\" (UID: \"0e190a98-17fe-46f6-adcf-34223eaf820e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4"
Oct 30 00:56:52.287738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.286857    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e190a98-17fe-46f6-adcf-34223eaf820e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4\" (UID: \"0e190a98-17fe-46f6-adcf-34223eaf820e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4"
Oct 30 00:56:52.423789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.405240    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h45xr\" (UniqueName: \"kubernetes.io/projected/0e190a98-17fe-46f6-adcf-34223eaf820e-kube-api-access-h45xr\") pod \"payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4\" (UID: \"0e190a98-17fe-46f6-adcf-34223eaf820e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4"
Oct 30 00:56:52.423789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.405298    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e190a98-17fe-46f6-adcf-34223eaf820e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4\" (UID: \"0e190a98-17fe-46f6-adcf-34223eaf820e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4"
Oct 30 00:56:52.427512 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.426900    6099 patch_prober.go:29] interesting pod/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 00:56:52.427512 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.426964    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj" podUID="e82201b7-f285-4ec3-ba2c-ddb29d770391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 00:56:52.735306 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.735214    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c3dff8e-63ac-43bc-be66-c53b21a1ceb2" path="/var/lib/kubelet/pods/5c3dff8e-63ac-43bc-be66-c53b21a1ceb2/volumes"
Oct 30 00:56:52.736469 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.736443    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4"]
Oct 30 00:56:52.757180 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.757157    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a32412\""
Oct 30 00:56:52.819731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.803229    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e190a98-17fe-46f6-adcf-34223eaf820e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4\" (UID: \"0e190a98-17fe-46f6-adcf-34223eaf820e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4"
Oct 30 00:56:52.819731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.803885    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h45xr\" (UniqueName: \"kubernetes.io/projected/0e190a98-17fe-46f6-adcf-34223eaf820e-kube-api-access-h45xr\") pod \"payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4\" (UID: \"0e190a98-17fe-46f6-adcf-34223eaf820e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4"
Oct 30 00:56:52.852893 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.852854    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m"]
Oct 30 00:56:52.855533 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.854805    6099 patch_prober.go:29] interesting pod/e2e-api-a-353403-openshift-default-5f567587b9-q56px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 00:56:52.855533 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.854855    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px" podUID="4a920d38-7878-4299-b399-64f3f76d7e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 00:56:52.894459 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.894418    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4"
Oct 30 00:56:52.921317 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.921282    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m"
Oct 30 00:56:52.942661 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.930662    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m"]
Oct 30 00:56:52.959588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.959556    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72" event={"ID":"bdf12444-9b7c-4721-900b-44b11df44dd1","Type":"ContainerStarted","Data":"cc1b96ec9b18656aeac10b69e06acc1438e7595307bb741bb3a5ef4ab8072137"}
Oct 30 00:56:52.999768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.994879    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20857824-d001-4494-97f3-ff58b868c3c2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a32412-ccf56585d-ns54m\" (UID: \"20857824-d001-4494-97f3-ff58b868c3c2\") " pod="openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m"
Oct 30 00:56:52.999768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:52.995227    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfxhg\" (UniqueName: \"kubernetes.io/projected/20857824-d001-4494-97f3-ff58b868c3c2-kube-api-access-kfxhg\") pod \"payload-processing-e2e-api-b-a32412-ccf56585d-ns54m\" (UID: \"20857824-d001-4494-97f3-ff58b868c3c2\") " pod="openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m"
Oct 30 00:56:53.051183 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.051059    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:56:53.062967 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.062924    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 30 00:56:53.064016 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.063977    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72" podUID="bdf12444-9b7c-4721-900b-44b11df44dd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 30 00:56:53.124892 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.124848    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfxhg\" (UniqueName: \"kubernetes.io/projected/20857824-d001-4494-97f3-ff58b868c3c2-kube-api-access-kfxhg\") pod \"payload-processing-e2e-api-b-a32412-ccf56585d-ns54m\" (UID: \"20857824-d001-4494-97f3-ff58b868c3c2\") " pod="openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m"
Oct 30 00:56:53.126864 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.126834    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20857824-d001-4494-97f3-ff58b868c3c2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a32412-ccf56585d-ns54m\" (UID: \"20857824-d001-4494-97f3-ff58b868c3c2\") " pod="openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m"
Oct 30 00:56:53.129062 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.128173    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20857824-d001-4494-97f3-ff58b868c3c2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a32412-ccf56585d-ns54m\" (UID: \"20857824-d001-4494-97f3-ff58b868c3c2\") " pod="openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m"
Oct 30 00:56:53.155034 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.154998    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfxhg\" (UniqueName: \"kubernetes.io/projected/20857824-d001-4494-97f3-ff58b868c3c2-kube-api-access-kfxhg\") pod \"payload-processing-e2e-api-b-a32412-ccf56585d-ns54m\" (UID: \"20857824-d001-4494-97f3-ff58b868c3c2\") " pod="openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m"
Oct 30 00:56:53.286600 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.286124    6099 patch_prober.go:29] interesting pod/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 00:56:53.286600 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.286187    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj" podUID="e82201b7-f285-4ec3-ba2c-ddb29d770391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 00:56:53.405071 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.353607    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m"
Oct 30 00:56:53.405071 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.403802    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"]
Oct 30 00:56:53.507116 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.491004    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"
Oct 30 00:56:53.508675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.508628    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254ea03c-844c-4295-aa4b-7b57cbe7efed-openshift-service-ca\") pod \"maas-api-e2e-api-b-73b32e-67b8b984f5-5txps\" (UID: \"254ea03c-844c-4295-aa4b-7b57cbe7efed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"
Oct 30 00:56:53.509145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.508918    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254ea03c-844c-4295-aa4b-7b57cbe7efed-maas-api-tls\") pod \"maas-api-e2e-api-b-73b32e-67b8b984f5-5txps\" (UID: \"254ea03c-844c-4295-aa4b-7b57cbe7efed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"
Oct 30 00:56:53.509445 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.509424    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr9pj\" (UniqueName: \"kubernetes.io/projected/254ea03c-844c-4295-aa4b-7b57cbe7efed-kube-api-access-hr9pj\") pod \"maas-api-e2e-api-b-73b32e-67b8b984f5-5txps\" (UID: \"254ea03c-844c-4295-aa4b-7b57cbe7efed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"
Oct 30 00:56:53.509647 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.509626    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254ea03c-844c-4295-aa4b-7b57cbe7efed-metrics-certs\") pod \"maas-api-e2e-api-b-73b32e-67b8b984f5-5txps\" (UID: \"254ea03c-844c-4295-aa4b-7b57cbe7efed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"
Oct 30 00:56:53.537457 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.515766    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-73b32e-serving-cert\""
Oct 30 00:56:53.537457 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.516521    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 00:56:53.537457 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.516618    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k5bc5\""
Oct 30 00:56:53.614751 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.613742    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr9pj\" (UniqueName: \"kubernetes.io/projected/254ea03c-844c-4295-aa4b-7b57cbe7efed-kube-api-access-hr9pj\") pod \"maas-api-e2e-api-b-73b32e-67b8b984f5-5txps\" (UID: \"254ea03c-844c-4295-aa4b-7b57cbe7efed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"
Oct 30 00:56:53.614751 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.613885    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254ea03c-844c-4295-aa4b-7b57cbe7efed-metrics-certs\") pod \"maas-api-e2e-api-b-73b32e-67b8b984f5-5txps\" (UID: \"254ea03c-844c-4295-aa4b-7b57cbe7efed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"
Oct 30 00:56:53.649733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.639016    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254ea03c-844c-4295-aa4b-7b57cbe7efed-openshift-service-ca\") pod \"maas-api-e2e-api-b-73b32e-67b8b984f5-5txps\" (UID: \"254ea03c-844c-4295-aa4b-7b57cbe7efed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"
Oct 30 00:56:53.649733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.639126    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254ea03c-844c-4295-aa4b-7b57cbe7efed-maas-api-tls\") pod \"maas-api-e2e-api-b-73b32e-67b8b984f5-5txps\" (UID: \"254ea03c-844c-4295-aa4b-7b57cbe7efed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"
Oct 30 00:56:53.718005 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.662280    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254ea03c-844c-4295-aa4b-7b57cbe7efed-openshift-service-ca\") pod \"maas-api-e2e-api-b-73b32e-67b8b984f5-5txps\" (UID: \"254ea03c-844c-4295-aa4b-7b57cbe7efed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"
Oct 30 00:56:53.718005 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.687783    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"]
Oct 30 00:56:53.755933 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.755866    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254ea03c-844c-4295-aa4b-7b57cbe7efed-metrics-certs\") pod \"maas-api-e2e-api-b-73b32e-67b8b984f5-5txps\" (UID: \"254ea03c-844c-4295-aa4b-7b57cbe7efed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"
Oct 30 00:56:53.756900 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.756851    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254ea03c-844c-4295-aa4b-7b57cbe7efed-maas-api-tls\") pod \"maas-api-e2e-api-b-73b32e-67b8b984f5-5txps\" (UID: \"254ea03c-844c-4295-aa4b-7b57cbe7efed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"
Oct 30 00:56:53.759593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.759508    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr9pj\" (UniqueName: \"kubernetes.io/projected/254ea03c-844c-4295-aa4b-7b57cbe7efed-kube-api-access-hr9pj\") pod \"maas-api-e2e-api-b-73b32e-67b8b984f5-5txps\" (UID: \"254ea03c-844c-4295-aa4b-7b57cbe7efed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"
Oct 30 00:56:53.844829 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.841606    6099 patch_prober.go:29] interesting pod/e2e-api-a-353403-openshift-default-5f567587b9-q56px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 00:56:53.844829 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.841669    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px" podUID="4a920d38-7878-4299-b399-64f3f76d7e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 00:56:53.949182 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:53.948916    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"
Oct 30 00:56:54.221127 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.195833    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 30 00:56:54.221127 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.195896    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72" podUID="bdf12444-9b7c-4721-900b-44b11df44dd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 30 00:56:54.227931 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.227832    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4"]
Oct 30 00:56:54.231659 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:56:54.231614    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e190a98_17fe_46f6_adcf_34223eaf820e.slice/crio-ac59ecf0ce90b52bd1f24fe98fb3c11961c07b66e0d7d9de08145af46595bdaf WatchSource:0}: Error finding container ac59ecf0ce90b52bd1f24fe98fb3c11961c07b66e0d7d9de08145af46595bdaf: Status 404 returned error can't find the container with id ac59ecf0ce90b52bd1f24fe98fb3c11961c07b66e0d7d9de08145af46595bdaf
Oct 30 00:56:54.237447 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.237090    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4"
Oct 30 00:56:54.301784 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.290723    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c744876-e4fc-4433-8bcd-7c490336f1da-kube-api-access-ksv4s\" (UniqueName: \"kubernetes.io/projected/7c744876-e4fc-4433-8bcd-7c490336f1da-kube-api-access-ksv4s\") pod \"7c744876-e4fc-4433-8bcd-7c490336f1da\" (UID: \"7c744876-e4fc-4433-8bcd-7c490336f1da\") "
Oct 30 00:56:54.301784 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.290817    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c744876-e4fc-4433-8bcd-7c490336f1da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c744876-e4fc-4433-8bcd-7c490336f1da-maas-api-openshift-ca\") pod \"7c744876-e4fc-4433-8bcd-7c490336f1da\" (UID: \"7c744876-e4fc-4433-8bcd-7c490336f1da\") "
Oct 30 00:56:54.330257 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.330209    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c744876-e4fc-4433-8bcd-7c490336f1da-maas-api-openshift-ca" pod "7c744876-e4fc-4433-8bcd-7c490336f1da" (UID: "7c744876-e4fc-4433-8bcd-7c490336f1da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:56:54.330901 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.330871    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c744876-e4fc-4433-8bcd-7c490336f1da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c744876-e4fc-4433-8bcd-7c490336f1da-maas-api-xks-ca\") pod \"7c744876-e4fc-4433-8bcd-7c490336f1da\" (UID: \"7c744876-e4fc-4433-8bcd-7c490336f1da\") "
Oct 30 00:56:54.333876 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.332976    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c744876-e4fc-4433-8bcd-7c490336f1da-kube-api-access-ksv4s" pod "7c744876-e4fc-4433-8bcd-7c490336f1da" (UID: "7c744876-e4fc-4433-8bcd-7c490336f1da"). InnerVolumeSpecName "kube-api-access-ksv4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:54.334656 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.334635    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c744876-e4fc-4433-8bcd-7c490336f1da-maas-api-xks-ca" pod "7c744876-e4fc-4433-8bcd-7c490336f1da" (UID: "7c744876-e4fc-4433-8bcd-7c490336f1da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:56:54.335807 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.335329    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c744876-e4fc-4433-8bcd-7c490336f1da-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:54.335807 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.335354    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksv4s\" (UniqueName: \"kubernetes.io/projected/7c744876-e4fc-4433-8bcd-7c490336f1da-kube-api-access-ksv4s\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:54.335807 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.335363    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c744876-e4fc-4433-8bcd-7c490336f1da-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:56:54.344751 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.344698    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:54.345815 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.345600    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:54.355733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.355656    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 00:56:54.438758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.438630    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m"]
Oct 30 00:56:54.881731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.853083    6099 patch_prober.go:29] interesting pod/e2e-api-a-353403-openshift-default-5f567587b9-q56px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 00:56:54.881731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:54.853144    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px" podUID="4a920d38-7878-4299-b399-64f3f76d7e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 00:56:55.054746 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:55.053421    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 30 00:56:55.054746 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:55.053471    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72" podUID="bdf12444-9b7c-4721-900b-44b11df44dd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 30 00:56:55.054746 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:56:55.054381    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254ea03c_844c_4295_aa4b_7b57cbe7efed.slice/crio-418d5cff240fc3652af63be38b1c77cd18065c388b34e5b2312aafc20e156e02 WatchSource:0}: Error finding container 418d5cff240fc3652af63be38b1c77cd18065c388b34e5b2312aafc20e156e02: Status 404 returned error can't find the container with id 418d5cff240fc3652af63be38b1c77cd18065c388b34e5b2312aafc20e156e02
Oct 30 00:56:55.056365 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:55.056342    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"]
Oct 30 00:56:55.254206 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:55.250987    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m" event={"ID":"20857824-d001-4494-97f3-ff58b868c3c2","Type":"ContainerStarted","Data":"21b2b68e62de806c328d1946643254519cab3e76f988257b0e72908f9ab25477"}
Oct 30 00:56:55.254206 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:55.251053    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m" event={"ID":"20857824-d001-4494-97f3-ff58b868c3c2","Type":"ContainerStarted","Data":"6ff55f112fc5499244152c6694cf55212cc4844ab1b85f0cb3b8fe3c6931cf74"}
Oct 30 00:56:55.254614 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:55.254574    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m"
Oct 30 00:56:55.261387 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:55.260549    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4" event={"ID":"0e190a98-17fe-46f6-adcf-34223eaf820e","Type":"ContainerStarted","Data":"28a44dc7433d5dd5158e619d7485433c6e041166705909aea99f6cb0bec6ea48"}
Oct 30 00:56:55.261387 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:55.260599    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4" event={"ID":"0e190a98-17fe-46f6-adcf-34223eaf820e","Type":"ContainerStarted","Data":"ac59ecf0ce90b52bd1f24fe98fb3c11961c07b66e0d7d9de08145af46595bdaf"}
Oct 30 00:56:55.261387 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:55.260804    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4"
Oct 30 00:56:55.362837 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:55.360642    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4" event={"ID":"7c744876-e4fc-4433-8bcd-7c490336f1da","Type":"ContainerDied","Data":"baf95a506a3e15add2a057e13bd72bb0b555fea76fd936b084852b6986282c57"}
Oct 30 00:56:55.363182 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:55.363161    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baf95a506a3e15add2a057e13bd72bb0b555fea76fd936b084852b6986282c57"
Oct 30 00:56:55.363310 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:55.362781    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4"
Oct 30 00:56:55.383436 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:55.383353    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps" event={"ID":"254ea03c-844c-4295-aa4b-7b57cbe7efed","Type":"ContainerStarted","Data":"418d5cff240fc3652af63be38b1c77cd18065c388b34e5b2312aafc20e156e02"}
Oct 30 00:56:55.449736 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:55.449660    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m" podStartSLOduration=3.449642227 podStartE2EDuration="3.449642227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:55.324864981 +0000 UTC m=+1309.181772734" watchObservedRunningTime="2026-10-30 00:56:55.449642227 +0000 UTC m=+1309.306549977"
Oct 30 00:56:55.463547 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:55.463497    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4" podStartSLOduration=3.46347834 podStartE2EDuration="3.46347834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:55.383297956 +0000 UTC m=+1309.240205711" watchObservedRunningTime="2026-10-30 00:56:55.46347834 +0000 UTC m=+1309.320386094"
Oct 30 00:56:55.838968 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:55.838610    6099 patch_prober.go:29] interesting pod/e2e-api-a-353403-openshift-default-5f567587b9-q56px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 00:56:55.838968 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:55.838659    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px" podUID="4a920d38-7878-4299-b399-64f3f76d7e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 00:56:56.035690 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:56.035652    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j"]
Oct 30 00:56:56.040204 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:56.040175    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce23d71-pjq5j"]
Oct 30 00:56:56.051135 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:56.050798    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 30 00:56:56.051135 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:56.050852    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72" podUID="bdf12444-9b7c-4721-900b-44b11df44dd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 30 00:56:56.316731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:56.314275    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed962bb-ac51-4b94-9443-2661169d35fe" path="/var/lib/kubelet/pods/7ed962bb-ac51-4b94-9443-2661169d35fe/volumes"
Oct 30 00:56:56.833942 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:56.833897    6099 patch_prober.go:29] interesting pod/e2e-api-a-353403-openshift-default-5f567587b9-q56px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 00:56:56.834127 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:56.833966    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px" podUID="4a920d38-7878-4299-b399-64f3f76d7e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 00:56:57.074731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:57.057200    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 30 00:56:57.074731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:57.057259    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72" podUID="bdf12444-9b7c-4721-900b-44b11df44dd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 30 00:56:57.839926 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:57.839892    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:56:57.840650 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:57.840472    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:56:57.841413 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:57.841390    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 00:56:58.051218 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:58.051187    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 30 00:56:58.051414 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:58.051265    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72" podUID="bdf12444-9b7c-4721-900b-44b11df44dd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 30 00:56:59.364881 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:59.364838    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6"]
Oct 30 00:56:59.497525 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:59.496398    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:56:59.497525 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:59.497015    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:56:59.504911 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:56:59.504860    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 00:57:00.510369 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:00.510326    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps" event={"ID":"254ea03c-844c-4295-aa4b-7b57cbe7efed","Type":"ContainerStarted","Data":"3873b8b3b12c2553fa42e2827ace0123907174bb839eea5aa3abe71580f14304"}
Oct 30 00:57:00.510864 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:00.510430    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"
Oct 30 00:57:00.555741 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:00.555675    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps" podStartSLOduration=2.722899908 podStartE2EDuration="7.555661158s" totalImagesPullingTime="4.83276125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:00.55104561 +0000 UTC m=+1314.407953370" watchObservedRunningTime="2026-10-30 00:57:00.555661158 +0000 UTC m=+1314.412568912"
Oct 30 00:57:01.034945 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:01.034909    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj"]
Oct 30 00:57:01.039173 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:01.039150    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38441b21-g29wj"]
Oct 30 00:57:02.264507 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:02.264475    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fbc0473-def5-4b5b-b78b-7719d7a6e443" path="/var/lib/kubelet/pods/6fbc0473-def5-4b5b-b78b-7719d7a6e443/volumes"
Oct 30 00:57:04.365250 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:04.365178    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6" podUID="4d97ad33-c251-46b6-b0dd-90112f148297" containerName="payload-pre-processing" containerID="cri-o://847e6d93e2aef370f3525898a74ba98413587954f46b76cc9a44f7608f0874c4" gracePeriod=25
Oct 30 00:57:06.429208 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:06.429181    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4"
Oct 30 00:57:06.527279 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:06.527249    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"
Oct 30 00:57:07.441750 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:07.441671    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m"
Oct 30 00:57:08.792489 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:08.792453    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx"]
Oct 30 00:57:08.793312 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:08.793295    6099 cpu_manager.go:414] "containerMap: removing container" podUID="7c744876-e4fc-4433-8bcd-7c490336f1da" containerName="revoke-keys"
Oct 30 00:57:08.793370 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:08.793312    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c744876-e4fc-4433-8bcd-7c490336f1da" containerName="revoke-keys"
Oct 30 00:57:08.793452 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:08.793443    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c744876-e4fc-4433-8bcd-7c490336f1da" containerName="revoke-keys"
Oct 30 00:57:08.799020 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:08.798998    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx"
Oct 30 00:57:08.816582 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:08.816556    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx"]
Oct 30 00:57:08.822733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:08.822470    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:57:08.822864 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:08.822690    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:57:08.858480 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:08.858447    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98809b25-de5f-4cb7-add8-6a0aa323aa9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6aa0d68-cv5lx\" (UID: \"98809b25-de5f-4cb7-add8-6a0aa323aa9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx"
Oct 30 00:57:08.858675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:08.858585    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98809b25-de5f-4cb7-add8-6a0aa323aa9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6aa0d68-cv5lx\" (UID: \"98809b25-de5f-4cb7-add8-6a0aa323aa9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx"
Oct 30 00:57:08.858784 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:08.858673    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb5kp\" (UniqueName: \"kubernetes.io/projected/98809b25-de5f-4cb7-add8-6a0aa323aa9d-kube-api-access-vb5kp\") pod \"maas-api-revoke-sub-f6aa0d68-cv5lx\" (UID: \"98809b25-de5f-4cb7-add8-6a0aa323aa9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx"
Oct 30 00:57:08.989735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:08.987256    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98809b25-de5f-4cb7-add8-6a0aa323aa9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6aa0d68-cv5lx\" (UID: \"98809b25-de5f-4cb7-add8-6a0aa323aa9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx"
Oct 30 00:57:08.989735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:08.987339    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb5kp\" (UniqueName: \"kubernetes.io/projected/98809b25-de5f-4cb7-add8-6a0aa323aa9d-kube-api-access-vb5kp\") pod \"maas-api-revoke-sub-f6aa0d68-cv5lx\" (UID: \"98809b25-de5f-4cb7-add8-6a0aa323aa9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx"
Oct 30 00:57:08.989735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:08.987766    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98809b25-de5f-4cb7-add8-6a0aa323aa9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6aa0d68-cv5lx\" (UID: \"98809b25-de5f-4cb7-add8-6a0aa323aa9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx"
Oct 30 00:57:08.996544 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:08.996513    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98809b25-de5f-4cb7-add8-6a0aa323aa9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6aa0d68-cv5lx\" (UID: \"98809b25-de5f-4cb7-add8-6a0aa323aa9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx"
Oct 30 00:57:08.998299 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:08.998272    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98809b25-de5f-4cb7-add8-6a0aa323aa9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6aa0d68-cv5lx\" (UID: \"98809b25-de5f-4cb7-add8-6a0aa323aa9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx"
Oct 30 00:57:09.024379 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:09.024347    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb5kp\" (UniqueName: \"kubernetes.io/projected/98809b25-de5f-4cb7-add8-6a0aa323aa9d-kube-api-access-vb5kp\") pod \"maas-api-revoke-sub-f6aa0d68-cv5lx\" (UID: \"98809b25-de5f-4cb7-add8-6a0aa323aa9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx"
Oct 30 00:57:09.126028 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:09.125927    6099 scope.go:122] "RemoveContainer" containerID="446bfddbd15b1490cf4dc0c42f06616d8226420b831b3cd1c46d703e172d8a9d"
Oct 30 00:57:09.128492 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:09.128466    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx"
Oct 30 00:57:09.153318 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:09.153187    6099 scope.go:122] "RemoveContainer" containerID="641cff1378937e8d9697f30b0a84dfe9cf0dfe8922058658e5d843686bead7dc"
Oct 30 00:57:09.185852 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:09.185825    6099 scope.go:122] "RemoveContainer" containerID="b3bc3a4dafc51fe6434186b8a79191a33c26736ad36ea1201968debfff73961a"
Oct 30 00:57:09.200222 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:09.200189    6099 scope.go:122] "RemoveContainer" containerID="30e0c992865527ccb833c293d94144337463c0b50cabb83eabfdb4947f8dae01"
Oct 30 00:57:09.278470 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:09.278402    6099 scope.go:122] "RemoveContainer" containerID="9f884e8d88195a356f55278d662e70abc75f4e05a0c0074d58a07f7ca6532678"
Oct 30 00:57:09.338769 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:09.325133    6099 scope.go:122] "RemoveContainer" containerID="e9c1aa64b232d640149da1c49f092b722435cb8486a0e9a959d8be2782492699"
Oct 30 00:57:09.344782 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:09.343868    6099 scope.go:122] "RemoveContainer" containerID="fea4b2c21da224ed79b3328d8f021273a4be073e20c995eec6febfa6e971e913"
Oct 30 00:57:09.403431 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:09.402409    6099 scope.go:122] "RemoveContainer" containerID="4dee8ac89c37983f44b795560a9bf0599710b26a966f85235e6ed6df89ac4cdc"
Oct 30 00:57:09.448537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:09.448409    6099 scope.go:122] "RemoveContainer" containerID="5706c6a24e324265d6c19123e14716de9bf82d9af004ca6749a0adb4fad2b3e3"
Oct 30 00:57:09.537376 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:57:09.537289    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98809b25_de5f_4cb7_add8_6a0aa323aa9d.slice/crio-f7aeb9ea0b261c2f5b55304df0f86de7b507a48a459d2852a5a48487977bd814 WatchSource:0}: Error finding container f7aeb9ea0b261c2f5b55304df0f86de7b507a48a459d2852a5a48487977bd814: Status 404 returned error can't find the container with id f7aeb9ea0b261c2f5b55304df0f86de7b507a48a459d2852a5a48487977bd814
Oct 30 00:57:09.538285 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:09.538262    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx"]
Oct 30 00:57:09.642433 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:09.642398    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx" event={"ID":"98809b25-de5f-4cb7-add8-6a0aa323aa9d","Type":"ContainerStarted","Data":"f7aeb9ea0b261c2f5b55304df0f86de7b507a48a459d2852a5a48487977bd814"}
Oct 30 00:57:10.553374 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.553340    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"]
Oct 30 00:57:10.558657 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.558632    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.591945 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.591919    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"]
Oct 30 00:57:10.599024 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.599000    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-601aea-openshift-default-dockercfg-cjn6r\""
Oct 30 00:57:10.629008 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.628967    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-podinfo\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.629141 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.629038    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94blq\" (UniqueName: \"kubernetes.io/projected/90b346c4-cc85-4fd7-9578-0533e8e902a9-kube-api-access-94blq\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.629141 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.629085    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-workload-certs\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.629141 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.629118    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-envoy\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.629250 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.629149    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b346c4-cc85-4fd7-9578-0533e8e902a9-istiod-ca-cert\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.629385 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.629363    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-credential-socket\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.629420 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.629395    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-token\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.629474 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.629457    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-workload-socket\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.629510 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.629491    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-data\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.734728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.733892    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b346c4-cc85-4fd7-9578-0533e8e902a9-istiod-ca-cert\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.734728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.734296    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-credential-socket\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.734728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.734332    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-token\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.734728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.734386    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-workload-socket\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.734728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.734413    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-data\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.734728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.734514    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-podinfo\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.734728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.734639    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94blq\" (UniqueName: \"kubernetes.io/projected/90b346c4-cc85-4fd7-9578-0533e8e902a9-kube-api-access-94blq\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.734728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.734683    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-workload-certs\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.734728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.734733    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-envoy\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.742731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.740313    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-workload-socket\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.742731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.741022    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b346c4-cc85-4fd7-9578-0533e8e902a9-istiod-ca-cert\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.742731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.741271    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-credential-socket\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.746725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.745030    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-data\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.746725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.745568    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-workload-certs\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.755727 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.751099    6099 generic.go:356] "Generic (PLEG): container finished" podID="98809b25-de5f-4cb7-add8-6a0aa323aa9d" containerID="43864f08ada5a10e5a6374101fd98d1f6203ed4faa2044b2062ed0f5214433d5" exitCode=0
Oct 30 00:57:10.755727 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.751490    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx" event={"ID":"98809b25-de5f-4cb7-add8-6a0aa323aa9d","Type":"ContainerDied","Data":"43864f08ada5a10e5a6374101fd98d1f6203ed4faa2044b2062ed0f5214433d5"}
Oct 30 00:57:10.763727 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.755943    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-envoy\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.786507 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.781395    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-token\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.801308 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.796790    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94blq\" (UniqueName: \"kubernetes.io/projected/90b346c4-cc85-4fd7-9578-0533e8e902a9-kube-api-access-94blq\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:10.801308 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:10.797281    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-podinfo\") pod \"e2e-api-b-601aea-openshift-default-8559b84459-xs799\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") " pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:11.014208 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:11.014108    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:11.019126 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:11.018922    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6" podUID="4d97ad33-c251-46b6-b0dd-90112f148297" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:57:11.352428 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:57:11.352394    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b346c4_cc85_4fd7_9578_0533e8e902a9.slice/crio-700e552c258fecf92aab6a9c78fa9f16a79571b3593e63853bf128da32adf647 WatchSource:0}: Error finding container 700e552c258fecf92aab6a9c78fa9f16a79571b3593e63853bf128da32adf647: Status 404 returned error can't find the container with id 700e552c258fecf92aab6a9c78fa9f16a79571b3593e63853bf128da32adf647
Oct 30 00:57:11.354194 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:11.354166    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"]
Oct 30 00:57:11.356393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:11.356355    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:11.356576 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:11.356554    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:11.356744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:11.356699    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:11.762524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:11.762445    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799" event={"ID":"90b346c4-cc85-4fd7-9578-0533e8e902a9","Type":"ContainerStarted","Data":"1652164599eab84070cc7219bc6df07fdc449c1c9cfa7661091ff2acc9113ac7"}
Oct 30 00:57:11.762524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:11.762498    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799" event={"ID":"90b346c4-cc85-4fd7-9578-0533e8e902a9","Type":"ContainerStarted","Data":"700e552c258fecf92aab6a9c78fa9f16a79571b3593e63853bf128da32adf647"}
Oct 30 00:57:11.810619 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:11.808490    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799" podStartSLOduration=1.808469968 podStartE2EDuration="1.808469968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:11.806770237 +0000 UTC m=+1325.663677984" watchObservedRunningTime="2026-10-30 00:57:11.808469968 +0000 UTC m=+1325.665377725"
Oct 30 00:57:11.978729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:11.975638    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx"
Oct 30 00:57:12.017107 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:12.015662    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:12.025660 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:12.025372    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:12.077619 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:12.075606    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98809b25-de5f-4cb7-add8-6a0aa323aa9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98809b25-de5f-4cb7-add8-6a0aa323aa9d-maas-api-xks-ca\") pod \"98809b25-de5f-4cb7-add8-6a0aa323aa9d\" (UID: \"98809b25-de5f-4cb7-add8-6a0aa323aa9d\") "
Oct 30 00:57:12.077826 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:12.077749    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98809b25-de5f-4cb7-add8-6a0aa323aa9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98809b25-de5f-4cb7-add8-6a0aa323aa9d-maas-api-openshift-ca\") pod \"98809b25-de5f-4cb7-add8-6a0aa323aa9d\" (UID: \"98809b25-de5f-4cb7-add8-6a0aa323aa9d\") "
Oct 30 00:57:12.078168 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:12.078140    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98809b25-de5f-4cb7-add8-6a0aa323aa9d-maas-api-openshift-ca" pod "98809b25-de5f-4cb7-add8-6a0aa323aa9d" (UID: "98809b25-de5f-4cb7-add8-6a0aa323aa9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:57:12.078278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:12.078264    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98809b25-de5f-4cb7-add8-6a0aa323aa9d-kube-api-access-vb5kp\" (UniqueName: \"kubernetes.io/projected/98809b25-de5f-4cb7-add8-6a0aa323aa9d-kube-api-access-vb5kp\") pod \"98809b25-de5f-4cb7-add8-6a0aa323aa9d\" (UID: \"98809b25-de5f-4cb7-add8-6a0aa323aa9d\") "
Oct 30 00:57:12.079778 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:12.079732    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98809b25-de5f-4cb7-add8-6a0aa323aa9d-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:57:12.082668 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:12.082563    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98809b25-de5f-4cb7-add8-6a0aa323aa9d-kube-api-access-vb5kp" pod "98809b25-de5f-4cb7-add8-6a0aa323aa9d" (UID: "98809b25-de5f-4cb7-add8-6a0aa323aa9d"). InnerVolumeSpecName "kube-api-access-vb5kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:57:12.085665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:12.085631    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98809b25-de5f-4cb7-add8-6a0aa323aa9d-maas-api-xks-ca" pod "98809b25-de5f-4cb7-add8-6a0aa323aa9d" (UID: "98809b25-de5f-4cb7-add8-6a0aa323aa9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:57:12.182105 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:12.182062    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98809b25-de5f-4cb7-add8-6a0aa323aa9d-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:57:12.182105 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:12.182101    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb5kp\" (UniqueName: \"kubernetes.io/projected/98809b25-de5f-4cb7-add8-6a0aa323aa9d-kube-api-access-vb5kp\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:57:12.775952 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:12.775905    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx"
Oct 30 00:57:12.776403 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:12.775917    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx" event={"ID":"98809b25-de5f-4cb7-add8-6a0aa323aa9d","Type":"ContainerDied","Data":"f7aeb9ea0b261c2f5b55304df0f86de7b507a48a459d2852a5a48487977bd814"}
Oct 30 00:57:12.776403 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:12.776056    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7aeb9ea0b261c2f5b55304df0f86de7b507a48a459d2852a5a48487977bd814"
Oct 30 00:57:12.776586 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:12.776557    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:12.777781 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:12.777760    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:57:13.865064 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:13.865024    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:57:15.041857 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:15.041818    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg"]
Oct 30 00:57:15.046257 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:15.046227    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64f60dc1-ckpzg"]
Oct 30 00:57:16.358729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:16.358556    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="740d18f1-010f-4859-96a3-c65d1dba7abf" path="/var/lib/kubelet/pods/740d18f1-010f-4859-96a3-c65d1dba7abf/volumes"
Oct 30 00:57:16.721130 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:57:16.721023    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:57:17.445730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:17.444722    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq"]
Oct 30 00:57:17.518113 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:17.455046    6099 cpu_manager.go:414] "containerMap: removing container" podUID="98809b25-de5f-4cb7-add8-6a0aa323aa9d" containerName="revoke-keys"
Oct 30 00:57:17.518113 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:17.455081    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98809b25-de5f-4cb7-add8-6a0aa323aa9d" containerName="revoke-keys"
Oct 30 00:57:17.518113 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:17.455681    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="98809b25-de5f-4cb7-add8-6a0aa323aa9d" containerName="revoke-keys"
Oct 30 00:57:17.915734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:17.908626    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:18.062603 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.062556    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:18.126721 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.095659    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:18.279825 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.278986    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq"
Oct 30 00:57:18.331759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.325811    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:18.356336 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.356299    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-601aea-dockercfg-pxct2\""
Oct 30 00:57:18.359822 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.356629    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-601aea\""
Oct 30 00:57:18.359990 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.359858    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": context deadline exceeded"
Oct 30 00:57:18.362752 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.361429    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq\" (UID: \"4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq"
Oct 30 00:57:18.362752 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.362404    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zlbj\" (UniqueName: \"kubernetes.io/projected/4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a-kube-api-access-4zlbj\") pod \"payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq\" (UID: \"4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq"
Oct 30 00:57:18.367738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.364154    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:18.367738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.366431    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 00:57:18.459912 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.459865    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:18.464241 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.464208    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:57:18.492316 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.464435    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:18.492316 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.464493    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:18.501724 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.494433    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq\" (UID: \"4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq"
Oct 30 00:57:18.547787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.536390    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zlbj\" (UniqueName: \"kubernetes.io/projected/4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a-kube-api-access-4zlbj\") pod \"payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq\" (UID: \"4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq"
Oct 30 00:57:18.547787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.520798    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:18.547787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.502877    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq\" (UID: \"4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq"
Oct 30 00:57:18.547787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.531825    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:18.547787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.521260    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded"
Oct 30 00:57:18.623326 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.623296    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq"]
Oct 30 00:57:18.623588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.623575    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg"]
Oct 30 00:57:18.666917 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.666883    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg"]
Oct 30 00:57:18.690837 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.667188    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg"
Oct 30 00:57:18.790789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.790757    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zlbj\" (UniqueName: \"kubernetes.io/projected/4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a-kube-api-access-4zlbj\") pod \"payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq\" (UID: \"4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq"
Oct 30 00:57:18.857674 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.856905    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:18.903498 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.903459    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f260864-36c4-4a59-b3a9-b47332908825-plugins-config-volume\") pod \"payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg\" (UID: \"4f260864-36c4-4a59-b3a9-b47332908825\") " pod="openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg"
Oct 30 00:57:18.903820 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.903793    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7qj2\" (UniqueName: \"kubernetes.io/projected/4f260864-36c4-4a59-b3a9-b47332908825-kube-api-access-z7qj2\") pod \"payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg\" (UID: \"4f260864-36c4-4a59-b3a9-b47332908825\") " pod="openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg"
Oct 30 00:57:18.943515 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:18.943313    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq"
Oct 30 00:57:19.012464 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:19.012415    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f260864-36c4-4a59-b3a9-b47332908825-plugins-config-volume\") pod \"payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg\" (UID: \"4f260864-36c4-4a59-b3a9-b47332908825\") " pod="openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg"
Oct 30 00:57:19.019866 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:19.019831    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7qj2\" (UniqueName: \"kubernetes.io/projected/4f260864-36c4-4a59-b3a9-b47332908825-kube-api-access-z7qj2\") pod \"payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg\" (UID: \"4f260864-36c4-4a59-b3a9-b47332908825\") " pod="openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg"
Oct 30 00:57:19.020560 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:19.017075    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f260864-36c4-4a59-b3a9-b47332908825-plugins-config-volume\") pod \"payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg\" (UID: \"4f260864-36c4-4a59-b3a9-b47332908825\") " pod="openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg"
Oct 30 00:57:19.064445 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:19.064410    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7qj2\" (UniqueName: \"kubernetes.io/projected/4f260864-36c4-4a59-b3a9-b47332908825-kube-api-access-z7qj2\") pod \"payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg\" (UID: \"4f260864-36c4-4a59-b3a9-b47332908825\") " pod="openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg"
Oct 30 00:57:19.280289 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:19.279991    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg"
Oct 30 00:57:19.393914 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:19.374687    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": context deadline exceeded"
Oct 30 00:57:19.393914 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:19.375804    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:57:19.584146 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:19.561610    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:57:19.584146 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:19.581236    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:19.584146 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:19.581753    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:57:19.837338 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:19.837219    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:20.450171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:20.450140    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:57:20.468867 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:57:20.468831    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice/crio-7e2678fa240ef4837699a8f6bae1f5cfbaa1664f174092e6001d8be60e30580b WatchSource:0}: Error finding container 7e2678fa240ef4837699a8f6bae1f5cfbaa1664f174092e6001d8be60e30580b: Status 404 returned error can't find the container with id 7e2678fa240ef4837699a8f6bae1f5cfbaa1664f174092e6001d8be60e30580b
Oct 30 00:57:20.510731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:20.489800    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn"]
Oct 30 00:57:20.555796 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:20.554515    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adee9bd3-jshpn"]
Oct 30 00:57:20.562531 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:20.560771    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq"]
Oct 30 00:57:21.153837 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.150479    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq"]
Oct 30 00:57:21.158261 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.158224    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6" podUID="4d97ad33-c251-46b6-b0dd-90112f148297" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:57:21.260628 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.248429    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq"
Oct 30 00:57:21.262491 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.260228    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq"]
Oct 30 00:57:21.293869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.293326    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:57:21.302863 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.294835    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:57:21.346518 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.344093    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8653d44e-bb96-412b-9d37-65e7f4a64064-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec2ef568-k7dwq\" (UID: \"8653d44e-bb96-412b-9d37-65e7f4a64064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq"
Oct 30 00:57:21.346518 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.344220    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-848lp\" (UniqueName: \"kubernetes.io/projected/8653d44e-bb96-412b-9d37-65e7f4a64064-kube-api-access-848lp\") pod \"maas-api-revoke-sub-ec2ef568-k7dwq\" (UID: \"8653d44e-bb96-412b-9d37-65e7f4a64064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq"
Oct 30 00:57:21.346518 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.344572    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8653d44e-bb96-412b-9d37-65e7f4a64064-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec2ef568-k7dwq\" (UID: \"8653d44e-bb96-412b-9d37-65e7f4a64064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq"
Oct 30 00:57:21.475404 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.475364    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8653d44e-bb96-412b-9d37-65e7f4a64064-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec2ef568-k7dwq\" (UID: \"8653d44e-bb96-412b-9d37-65e7f4a64064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq"
Oct 30 00:57:21.475999 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.475967    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8653d44e-bb96-412b-9d37-65e7f4a64064-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec2ef568-k7dwq\" (UID: \"8653d44e-bb96-412b-9d37-65e7f4a64064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq"
Oct 30 00:57:21.486353 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.476316    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-848lp\" (UniqueName: \"kubernetes.io/projected/8653d44e-bb96-412b-9d37-65e7f4a64064-kube-api-access-848lp\") pod \"maas-api-revoke-sub-ec2ef568-k7dwq\" (UID: \"8653d44e-bb96-412b-9d37-65e7f4a64064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq"
Oct 30 00:57:21.547732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.488558    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8653d44e-bb96-412b-9d37-65e7f4a64064-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec2ef568-k7dwq\" (UID: \"8653d44e-bb96-412b-9d37-65e7f4a64064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq"
Oct 30 00:57:21.548459 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.548431    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg"]
Oct 30 00:57:21.616760 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.609201    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-848lp\" (UniqueName: \"kubernetes.io/projected/8653d44e-bb96-412b-9d37-65e7f4a64064-kube-api-access-848lp\") pod \"maas-api-revoke-sub-ec2ef568-k7dwq\" (UID: \"8653d44e-bb96-412b-9d37-65e7f4a64064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq"
Oct 30 00:57:21.650125 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.631179    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8653d44e-bb96-412b-9d37-65e7f4a64064-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec2ef568-k7dwq\" (UID: \"8653d44e-bb96-412b-9d37-65e7f4a64064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq"
Oct 30 00:57:21.717733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.709757    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq"
Oct 30 00:57:21.907772 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.896538    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg" event={"ID":"4f260864-36c4-4a59-b3a9-b47332908825","Type":"ContainerStarted","Data":"282b21726f130a621a472ee44fd75b56172105a5a97847b7a5cb1803bcbfc181"}
Oct 30 00:57:21.947456 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.930732    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq" event={"ID":"4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a","Type":"ContainerStarted","Data":"3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5"}
Oct 30 00:57:21.948029 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.947992    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq" event={"ID":"4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a","Type":"ContainerStarted","Data":"7e2678fa240ef4837699a8f6bae1f5cfbaa1664f174092e6001d8be60e30580b"}
Oct 30 00:57:21.948210 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:21.948196    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq"
Oct 30 00:57:22.042728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:22.042407    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq" podStartSLOduration=5.042385243 podStartE2EDuration="5.042385243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:22.017599989 +0000 UTC m=+1335.874507745" watchObservedRunningTime="2026-10-30 00:57:22.042385243 +0000 UTC m=+1335.899293001"
Oct 30 00:57:22.437267 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:22.437231    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="533b61f4-794e-48b8-aa70-4dd498211e12" path="/var/lib/kubelet/pods/533b61f4-794e-48b8-aa70-4dd498211e12/volumes"
Oct 30 00:57:22.646579 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:22.646550    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq"]
Oct 30 00:57:22.651161 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:57:22.651135    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8653d44e_bb96_412b_9d37_65e7f4a64064.slice/crio-34222b85587024ad8a538f9bcaf3758bb7177f5534908336b6e88fc60c50231a WatchSource:0}: Error finding container 34222b85587024ad8a538f9bcaf3758bb7177f5534908336b6e88fc60c50231a: Status 404 returned error can't find the container with id 34222b85587024ad8a538f9bcaf3758bb7177f5534908336b6e88fc60c50231a
Oct 30 00:57:22.946185 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:22.945696    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg" event={"ID":"4f260864-36c4-4a59-b3a9-b47332908825","Type":"ContainerStarted","Data":"1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af"}
Oct 30 00:57:22.946396 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:22.946232    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg"
Oct 30 00:57:22.948037 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:22.947988    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq" event={"ID":"8653d44e-bb96-412b-9d37-65e7f4a64064","Type":"ContainerStarted","Data":"34222b85587024ad8a538f9bcaf3758bb7177f5534908336b6e88fc60c50231a"}
Oct 30 00:57:22.991835 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:22.989387    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg" podStartSLOduration=5.989327119 podStartE2EDuration="5.989327119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:22.987599556 +0000 UTC m=+1336.844507312" watchObservedRunningTime="2026-10-30 00:57:22.989327119 +0000 UTC m=+1336.846234876"
Oct 30 00:57:23.840201 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:23.839822    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:57:23.840201 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:23.839884    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx" podUID="e764760a-aad7-44be-9068-387287c0602c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:23.990236 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:23.990161    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq" event={"ID":"8653d44e-bb96-412b-9d37-65e7f4a64064","Type":"ContainerStarted","Data":"2a39584e87f13ae722a2b9cb93dfa5393d7b2511db18f420af6e769dae1119b0"}
Oct 30 00:57:24.032874 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:24.031297    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq" podStartSLOduration=4.031279381 podStartE2EDuration="4.031279381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:24.030616829 +0000 UTC m=+1337.887524586" watchObservedRunningTime="2026-10-30 00:57:24.031279381 +0000 UTC m=+1337.888187138"
Oct 30 00:57:24.779933 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:24.779904    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6"
Oct 30 00:57:24.794818 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:24.794775    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d97ad33-c251-46b6-b0dd-90112f148297-kube-api-access-xrlbx\" (UniqueName: \"kubernetes.io/projected/4d97ad33-c251-46b6-b0dd-90112f148297-kube-api-access-xrlbx\") pod \"4d97ad33-c251-46b6-b0dd-90112f148297\" (UID: \"4d97ad33-c251-46b6-b0dd-90112f148297\") "
Oct 30 00:57:24.795012 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:24.794833    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d97ad33-c251-46b6-b0dd-90112f148297-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d97ad33-c251-46b6-b0dd-90112f148297-plugins-config-volume\") pod \"4d97ad33-c251-46b6-b0dd-90112f148297\" (UID: \"4d97ad33-c251-46b6-b0dd-90112f148297\") "
Oct 30 00:57:24.795666 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:24.795596    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d97ad33-c251-46b6-b0dd-90112f148297-plugins-config-volume" pod "4d97ad33-c251-46b6-b0dd-90112f148297" (UID: "4d97ad33-c251-46b6-b0dd-90112f148297"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:57:24.795998 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:24.795976    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d97ad33-c251-46b6-b0dd-90112f148297-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:57:24.799936 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:24.799896    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d97ad33-c251-46b6-b0dd-90112f148297-kube-api-access-xrlbx" pod "4d97ad33-c251-46b6-b0dd-90112f148297" (UID: "4d97ad33-c251-46b6-b0dd-90112f148297"). InnerVolumeSpecName "kube-api-access-xrlbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:57:24.898291 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:24.898248    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrlbx\" (UniqueName: \"kubernetes.io/projected/4d97ad33-c251-46b6-b0dd-90112f148297-kube-api-access-xrlbx\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:57:25.095729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:25.095079    6099 generic.go:356] "Generic (PLEG): container finished" podID="4d97ad33-c251-46b6-b0dd-90112f148297" containerID="847e6d93e2aef370f3525898a74ba98413587954f46b76cc9a44f7608f0874c4" exitCode=0
Oct 30 00:57:25.095729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:25.095260    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6" event={"ID":"4d97ad33-c251-46b6-b0dd-90112f148297","Type":"ContainerDied","Data":"847e6d93e2aef370f3525898a74ba98413587954f46b76cc9a44f7608f0874c4"}
Oct 30 00:57:25.095729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:25.095311    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6" event={"ID":"4d97ad33-c251-46b6-b0dd-90112f148297","Type":"ContainerDied","Data":"cc9605c85c338160fe3eeda428ab3c71e35a49c9d3d413e9ff115bf40f496640"}
Oct 30 00:57:25.095729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:25.095354    6099 scope.go:122] "RemoveContainer" containerID="847e6d93e2aef370f3525898a74ba98413587954f46b76cc9a44f7608f0874c4"
Oct 30 00:57:25.095729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:25.095521    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6"
Oct 30 00:57:25.110898 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:25.106991    6099 generic.go:356] "Generic (PLEG): container finished" podID="8653d44e-bb96-412b-9d37-65e7f4a64064" containerID="2a39584e87f13ae722a2b9cb93dfa5393d7b2511db18f420af6e769dae1119b0" exitCode=0
Oct 30 00:57:25.110898 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:25.107288    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq" event={"ID":"8653d44e-bb96-412b-9d37-65e7f4a64064","Type":"ContainerDied","Data":"2a39584e87f13ae722a2b9cb93dfa5393d7b2511db18f420af6e769dae1119b0"}
Oct 30 00:57:25.143162 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:25.142829    6099 scope.go:122] "RemoveContainer" containerID="847e6d93e2aef370f3525898a74ba98413587954f46b76cc9a44f7608f0874c4"
Oct 30 00:57:25.143479 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:57:25.143383    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"847e6d93e2aef370f3525898a74ba98413587954f46b76cc9a44f7608f0874c4\": container with ID starting with 847e6d93e2aef370f3525898a74ba98413587954f46b76cc9a44f7608f0874c4 not found: ID does not exist" containerID="847e6d93e2aef370f3525898a74ba98413587954f46b76cc9a44f7608f0874c4"
Oct 30 00:57:25.143479 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:25.143421    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"847e6d93e2aef370f3525898a74ba98413587954f46b76cc9a44f7608f0874c4"} err="failed to get container status \"847e6d93e2aef370f3525898a74ba98413587954f46b76cc9a44f7608f0874c4\": rpc error: code = NotFound desc = could not find container \"847e6d93e2aef370f3525898a74ba98413587954f46b76cc9a44f7608f0874c4\": container with ID starting with 847e6d93e2aef370f3525898a74ba98413587954f46b76cc9a44f7608f0874c4 not found: ID does not exist"
Oct 30 00:57:25.275729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:25.271005    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6"]
Oct 30 00:57:25.278345 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:25.278302    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8090513-66c4b9c87-sdvw6"]
Oct 30 00:57:26.267131 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:26.267098    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d97ad33-c251-46b6-b0dd-90112f148297" path="/var/lib/kubelet/pods/4d97ad33-c251-46b6-b0dd-90112f148297/volumes"
Oct 30 00:57:27.216792 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:27.193446    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq" event={"ID":"8653d44e-bb96-412b-9d37-65e7f4a64064","Type":"ContainerDied","Data":"34222b85587024ad8a538f9bcaf3758bb7177f5534908336b6e88fc60c50231a"}
Oct 30 00:57:27.216964 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:27.216847    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34222b85587024ad8a538f9bcaf3758bb7177f5534908336b6e88fc60c50231a"
Oct 30 00:57:27.232746 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:27.232674    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq"
Oct 30 00:57:27.336700 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:27.336661    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8653d44e-bb96-412b-9d37-65e7f4a64064-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8653d44e-bb96-412b-9d37-65e7f4a64064-maas-api-openshift-ca\") pod \"8653d44e-bb96-412b-9d37-65e7f4a64064\" (UID: \"8653d44e-bb96-412b-9d37-65e7f4a64064\") "
Oct 30 00:57:27.337200 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:27.336859    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8653d44e-bb96-412b-9d37-65e7f4a64064-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8653d44e-bb96-412b-9d37-65e7f4a64064-maas-api-xks-ca\") pod \"8653d44e-bb96-412b-9d37-65e7f4a64064\" (UID: \"8653d44e-bb96-412b-9d37-65e7f4a64064\") "
Oct 30 00:57:27.337200 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:27.337127    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8653d44e-bb96-412b-9d37-65e7f4a64064-kube-api-access-848lp\" (UniqueName: \"kubernetes.io/projected/8653d44e-bb96-412b-9d37-65e7f4a64064-kube-api-access-848lp\") pod \"8653d44e-bb96-412b-9d37-65e7f4a64064\" (UID: \"8653d44e-bb96-412b-9d37-65e7f4a64064\") "
Oct 30 00:57:27.354198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:27.354160    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8653d44e-bb96-412b-9d37-65e7f4a64064-maas-api-openshift-ca" pod "8653d44e-bb96-412b-9d37-65e7f4a64064" (UID: "8653d44e-bb96-412b-9d37-65e7f4a64064"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:57:27.369743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:27.369675    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8653d44e-bb96-412b-9d37-65e7f4a64064-maas-api-xks-ca" pod "8653d44e-bb96-412b-9d37-65e7f4a64064" (UID: "8653d44e-bb96-412b-9d37-65e7f4a64064"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:57:27.369914 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:27.369885    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8653d44e-bb96-412b-9d37-65e7f4a64064-kube-api-access-848lp" pod "8653d44e-bb96-412b-9d37-65e7f4a64064" (UID: "8653d44e-bb96-412b-9d37-65e7f4a64064"). InnerVolumeSpecName "kube-api-access-848lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:57:27.440258 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:27.440223    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-848lp\" (UniqueName: \"kubernetes.io/projected/8653d44e-bb96-412b-9d37-65e7f4a64064-kube-api-access-848lp\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:57:27.440258 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:27.440255    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8653d44e-bb96-412b-9d37-65e7f4a64064-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:57:27.440258 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:27.440266    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8653d44e-bb96-412b-9d37-65e7f4a64064-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:57:28.142403 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:28.142365    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq"
Oct 30 00:57:30.992452 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:30.992395    6099 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:57:30.992916 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:30.992515    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw" podUID="19e9d22c-b2b4-4795-be11-ff2d98383a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:31.037591 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:31.037549    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt"]
Oct 30 00:57:31.050635 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:31.050602    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a55081f-m8bbt"]
Oct 30 00:57:32.294576 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:32.294535    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a438c487-876a-4b8b-87c2-2f13517e89eb" path="/var/lib/kubelet/pods/a438c487-876a-4b8b-87c2-2f13517e89eb/volumes"
Oct 30 00:57:32.952041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:32.952007    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq"
Oct 30 00:57:33.992455 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:33.992425    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg"
Oct 30 00:57:36.216571 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:36.216536    6099 patch_prober.go:29] interesting pod/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 00:57:36.217039 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:36.216590    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr" podUID="a9004633-73af-405e-8cd8-b1bd569602af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 00:57:36.241341 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:36.241304    6099 generic.go:356] "Generic (PLEG): container finished" podID="a9004633-73af-405e-8cd8-b1bd569602af" containerID="c843d64a739e4001754f1931db315e4e062563754e08d3d9a1b14440f895b27d" exitCode=0
Oct 30 00:57:36.241499 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:36.241371    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr" event={"ID":"a9004633-73af-405e-8cd8-b1bd569602af","Type":"ContainerDied","Data":"c843d64a739e4001754f1931db315e4e062563754e08d3d9a1b14440f895b27d"}
Oct 30 00:57:36.242059 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:36.242029    6099 scope.go:122] "RemoveContainer" containerID="c843d64a739e4001754f1931db315e4e062563754e08d3d9a1b14440f895b27d"
Oct 30 00:57:36.244396 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:36.244365    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:36.244480 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:36.244447    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:36.244525 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:36.244478    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:37.260578 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:37.260525    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr" event={"ID":"a9004633-73af-405e-8cd8-b1bd569602af","Type":"ContainerStarted","Data":"8df45b1fc6e1b6b783d73152c440ddb40e495192b42cf37333904d5061a90f36"}
Oct 30 00:57:37.801113 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:37.801077    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 00:57:37.801299 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:37.801150    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx" podUID="e764760a-aad7-44be-9068-387287c0602c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 00:57:37.812629 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:37.812595    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:57:37.812821 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:37.812784    6099 patch_prober.go:29] interesting pod/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 00:57:37.812885 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:37.812841    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr" podUID="a9004633-73af-405e-8cd8-b1bd569602af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 00:57:38.319885 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:38.319061    6099 generic.go:356] "Generic (PLEG): container finished" podID="19e9d22c-b2b4-4795-be11-ff2d98383a92" containerID="485ef597e39d43f5e309aec055ac6004c7ce23ed9b99c38bbb4b207671d7e6dc" exitCode=0
Oct 30 00:57:38.319885 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:38.319128    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw" event={"ID":"19e9d22c-b2b4-4795-be11-ff2d98383a92","Type":"ContainerDied","Data":"485ef597e39d43f5e309aec055ac6004c7ce23ed9b99c38bbb4b207671d7e6dc"}
Oct 30 00:57:38.319885 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:38.319834    6099 scope.go:122] "RemoveContainer" containerID="485ef597e39d43f5e309aec055ac6004c7ce23ed9b99c38bbb4b207671d7e6dc"
Oct 30 00:57:38.321184 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:38.321164    6099 generic.go:356] "Generic (PLEG): container finished" podID="e764760a-aad7-44be-9068-387287c0602c" containerID="4a3343d4a0da3b853c0b48dc5bb854a0fe56ba874dc6655339096fd7f6b7050a" exitCode=0
Oct 30 00:57:38.321288 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:38.321266    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx" event={"ID":"e764760a-aad7-44be-9068-387287c0602c","Type":"ContainerDied","Data":"4a3343d4a0da3b853c0b48dc5bb854a0fe56ba874dc6655339096fd7f6b7050a"}
Oct 30 00:57:38.321881 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:38.321864    6099 scope.go:122] "RemoveContainer" containerID="4a3343d4a0da3b853c0b48dc5bb854a0fe56ba874dc6655339096fd7f6b7050a"
Oct 30 00:57:38.322335 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:38.322302    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:38.322412 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:38.322385    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:38.322446 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:38.322431    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:38.323902 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:38.323874    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:38.323970 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:38.323954    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:38.324008 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:38.323999    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:38.816784 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:38.814805    6099 patch_prober.go:29] interesting pod/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 00:57:38.816784 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:38.814870    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr" podUID="a9004633-73af-405e-8cd8-b1bd569602af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 00:57:39.032947 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:39.032914    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9"]
Oct 30 00:57:39.037525 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:39.037495    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac1ae37-vrhv9"]
Oct 30 00:57:39.346551 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:39.346512    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw" event={"ID":"19e9d22c-b2b4-4795-be11-ff2d98383a92","Type":"ContainerStarted","Data":"8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853"}
Oct 30 00:57:39.351067 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:39.351029    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx" event={"ID":"e764760a-aad7-44be-9068-387287c0602c","Type":"ContainerStarted","Data":"d9fac4f2d3efb6e291959cba6511e86001b2117fba7e6efc6f7452822f6c4dcd"}
Oct 30 00:57:39.633809 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:39.629945    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:57:39.633809 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:39.630075    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 00:57:39.633809 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:39.630125    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx" podUID="e764760a-aad7-44be-9068-387287c0602c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 00:57:39.816161 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:39.816128    6099 patch_prober.go:29] interesting pod/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 00:57:39.816456 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:39.816426    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr" podUID="a9004633-73af-405e-8cd8-b1bd569602af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 00:57:40.242732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.235732    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:57:40.242732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.239201    6099 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 30 00:57:40.242732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.239249    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw" podUID="19e9d22c-b2b4-4795-be11-ff2d98383a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 30 00:57:40.263912 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.263876    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e6aab1-a6fd-4b49-b92d-7a7ab0982699" path="/var/lib/kubelet/pods/94e6aab1-a6fd-4b49-b92d-7a7ab0982699/volumes"
Oct 30 00:57:40.619631 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.614415    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 00:57:40.619631 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.614492    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx" podUID="e764760a-aad7-44be-9068-387287c0602c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 00:57:40.708797 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.708756    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt"]
Oct 30 00:57:40.781735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.711357    6099 cpu_manager.go:414] "containerMap: removing container" podUID="8653d44e-bb96-412b-9d37-65e7f4a64064" containerName="revoke-keys"
Oct 30 00:57:40.917970 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.917864    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8653d44e-bb96-412b-9d37-65e7f4a64064" containerName="revoke-keys"
Oct 30 00:57:40.922908 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.922869    6099 cpu_manager.go:414] "containerMap: removing container" podUID="4d97ad33-c251-46b6-b0dd-90112f148297" containerName="payload-pre-processing"
Oct 30 00:57:40.923148 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.923127    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d97ad33-c251-46b6-b0dd-90112f148297" containerName="payload-pre-processing"
Oct 30 00:57:40.948118 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.850392    6099 patch_prober.go:29] interesting pod/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 00:57:40.948476 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.948419    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr" podUID="a9004633-73af-405e-8cd8-b1bd569602af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 00:57:40.949853 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.949833    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="8653d44e-bb96-412b-9d37-65e7f4a64064" containerName="revoke-keys"
Oct 30 00:57:40.950033 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.950021    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d97ad33-c251-46b6-b0dd-90112f148297" containerName="payload-pre-processing"
Oct 30 00:57:40.958311 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.958211    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt"]
Oct 30 00:57:40.958437 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.958315    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt"
Oct 30 00:57:40.992738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.990038    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:57:40.992738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:40.990175    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:57:41.055837 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:41.055796    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6867b8-756jt\" (UID: \"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt"
Oct 30 00:57:41.056062 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:41.056039    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5gg2\" (UniqueName: \"kubernetes.io/projected/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-kube-api-access-v5gg2\") pod \"maas-api-revoke-keys-e2e-api-a-6867b8-756jt\" (UID: \"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt"
Oct 30 00:57:41.056215 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:41.056197    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6867b8-756jt\" (UID: \"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt"
Oct 30 00:57:41.157575 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:41.157532    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5gg2\" (UniqueName: \"kubernetes.io/projected/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-kube-api-access-v5gg2\") pod \"maas-api-revoke-keys-e2e-api-a-6867b8-756jt\" (UID: \"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt"
Oct 30 00:57:41.157836 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:41.157813    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6867b8-756jt\" (UID: \"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt"
Oct 30 00:57:41.157984 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:41.157965    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6867b8-756jt\" (UID: \"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt"
Oct 30 00:57:41.158684 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:41.158657    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6867b8-756jt\" (UID: \"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt"
Oct 30 00:57:41.161595 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:41.161572    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6867b8-756jt\" (UID: \"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt"
Oct 30 00:57:41.217780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:41.215338    6099 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 30 00:57:41.217780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:41.215417    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw" podUID="19e9d22c-b2b4-4795-be11-ff2d98383a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 30 00:57:41.247733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:41.247488    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5gg2\" (UniqueName: \"kubernetes.io/projected/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-kube-api-access-v5gg2\") pod \"maas-api-revoke-keys-e2e-api-a-6867b8-756jt\" (UID: \"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt"
Oct 30 00:57:41.355525 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:41.355488    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt"
Oct 30 00:57:41.644587 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:41.644545    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 00:57:41.645212 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:41.645176    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx" podUID="e764760a-aad7-44be-9068-387287c0602c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 00:57:41.835725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:41.831853    6099 patch_prober.go:29] interesting pod/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 00:57:41.835725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:41.831915    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr" podUID="a9004633-73af-405e-8cd8-b1bd569602af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 00:57:42.231787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:42.225121    6099 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 30 00:57:42.231787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:42.225186    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw" podUID="19e9d22c-b2b4-4795-be11-ff2d98383a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 30 00:57:42.453333 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:42.453299    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt"]
Oct 30 00:57:42.517573 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:57:42.517465    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce17e2f_7d6c_4491_b1ae_3043c7f8e87a.slice/crio-0a637a42f9f674a63c84564bc9a11b7762d1eeeb9384b32a570391b4952129d5 WatchSource:0}: Error finding container 0a637a42f9f674a63c84564bc9a11b7762d1eeeb9384b32a570391b4952129d5: Status 404 returned error can't find the container with id 0a637a42f9f674a63c84564bc9a11b7762d1eeeb9384b32a570391b4952129d5
Oct 30 00:57:42.615097 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:42.615045    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 00:57:42.615563 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:42.615374    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx" podUID="e764760a-aad7-44be-9068-387287c0602c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 00:57:42.856385 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:42.853401    6099 patch_prober.go:29] interesting pod/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 00:57:42.856385 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:42.853457    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr" podUID="a9004633-73af-405e-8cd8-b1bd569602af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 00:57:43.218783 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:43.218472    6099 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 30 00:57:43.218783 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:43.218541    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw" podUID="19e9d22c-b2b4-4795-be11-ff2d98383a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 30 00:57:43.618728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:43.616844    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 00:57:43.618728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:43.616890    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx" podUID="e764760a-aad7-44be-9068-387287c0602c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 00:57:43.629453 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:43.629352    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt" event={"ID":"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a","Type":"ContainerStarted","Data":"0c222070dd5cccb532cacf30b3b36ad69ff9050939c479ff8f768f23370a66cb"}
Oct 30 00:57:43.629453 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:43.629415    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt" event={"ID":"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a","Type":"ContainerStarted","Data":"0a637a42f9f674a63c84564bc9a11b7762d1eeeb9384b32a570391b4952129d5"}
Oct 30 00:57:43.707788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:43.706455    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt" podStartSLOduration=3.706432067 podStartE2EDuration="3.706432067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:43.689284813 +0000 UTC m=+1357.546192569" watchObservedRunningTime="2026-10-30 00:57:43.706432067 +0000 UTC m=+1357.563339823"
Oct 30 00:57:43.829740 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:43.828640    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:57:43.829740 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:43.829331    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:57:43.830541 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:43.830523    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:57:44.223731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:44.222756    6099 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 30 00:57:44.223731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:44.222837    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw" podUID="19e9d22c-b2b4-4795-be11-ff2d98383a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 30 00:57:44.619789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:44.619750    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 00:57:44.619997 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:44.619810    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx" podUID="e764760a-aad7-44be-9068-387287c0602c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 00:57:44.918651 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:44.918562    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:57:45.218953 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:45.218872    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:57:45.618614 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:45.618581    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:57:45.619125 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:45.619103    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:57:45.619865 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:45.619842    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:57:45.648120 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:45.648090    6099 generic.go:356] "Generic (PLEG): container finished" podID="fce17e2f-7d6c-4491-b1ae-3043c7f8e87a" containerID="0c222070dd5cccb532cacf30b3b36ad69ff9050939c479ff8f768f23370a66cb" exitCode=0
Oct 30 00:57:45.648235 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:45.648162    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt" event={"ID":"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a","Type":"ContainerDied","Data":"0c222070dd5cccb532cacf30b3b36ad69ff9050939c479ff8f768f23370a66cb"}
Oct 30 00:57:45.649334 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:45.649314    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 00:57:46.031372 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:46.031284    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7"]
Oct 30 00:57:46.036365 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:46.036322    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49070d8b-4znv7"]
Oct 30 00:57:46.265265 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:46.265233    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd1bd97-d7ff-44dd-88fd-bc4cfae19210" path="/var/lib/kubelet/pods/0fd1bd97-d7ff-44dd-88fd-bc4cfae19210/volumes"
Oct 30 00:57:46.671308 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:57:46.671268    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:57:46.832169 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:46.832143    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt"
Oct 30 00:57:46.922608 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:46.922518    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-maas-api-openshift-ca\") pod \"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a\" (UID: \"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a\") "
Oct 30 00:57:46.922818 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:46.922795    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-maas-api-xks-ca\") pod \"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a\" (UID: \"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a\") "
Oct 30 00:57:46.922900 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:46.922851    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-maas-api-openshift-ca" pod "fce17e2f-7d6c-4491-b1ae-3043c7f8e87a" (UID: "fce17e2f-7d6c-4491-b1ae-3043c7f8e87a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:57:46.923017 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:46.923002    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-kube-api-access-v5gg2\" (UniqueName: \"kubernetes.io/projected/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-kube-api-access-v5gg2\") pod \"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a\" (UID: \"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a\") "
Oct 30 00:57:46.924079 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:46.924049    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:57:46.925076 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:46.925052    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-maas-api-xks-ca" pod "fce17e2f-7d6c-4491-b1ae-3043c7f8e87a" (UID: "fce17e2f-7d6c-4491-b1ae-3043c7f8e87a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:57:46.925253 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:46.925236    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-kube-api-access-v5gg2" pod "fce17e2f-7d6c-4491-b1ae-3043c7f8e87a" (UID: "fce17e2f-7d6c-4491-b1ae-3043c7f8e87a"). InnerVolumeSpecName "kube-api-access-v5gg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:57:47.025984 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:47.025952    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:57:47.025984 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:47.025983    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5gg2\" (UniqueName: \"kubernetes.io/projected/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a-kube-api-access-v5gg2\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:57:47.671654 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:47.671620    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt"
Oct 30 00:57:47.672171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:47.671646    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt" event={"ID":"fce17e2f-7d6c-4491-b1ae-3043c7f8e87a","Type":"ContainerDied","Data":"0a637a42f9f674a63c84564bc9a11b7762d1eeeb9384b32a570391b4952129d5"}
Oct 30 00:57:47.672171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:47.671740    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a637a42f9f674a63c84564bc9a11b7762d1eeeb9384b32a570391b4952129d5"
Oct 30 00:57:51.039386 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:51.039353    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg"]
Oct 30 00:57:51.605107 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:51.605069    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:57:52.047753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:52.046224    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp"]
Oct 30 00:57:52.051546 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:52.051512    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74bb6183-lxxfp"]
Oct 30 00:57:52.266197 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:52.266164    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a853723b-1bf6-4458-ab94-056b85e63a31" path="/var/lib/kubelet/pods/a853723b-1bf6-4458-ab94-056b85e63a31/volumes"
Oct 30 00:57:54.507790 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:54.504700    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:56.054739 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:56.049936    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg" podUID="0519dba7-7145-4761-b18c-397c0bee13d8" containerName="payload-processing" containerID="cri-o://ec32240736c4e6473de34d4ef494e17744d70f751789fea6ed2d3c724a0dd2d2" gracePeriod=25
Oct 30 00:57:56.274178 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:56.270807    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:56.370198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:56.370095    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:57:56.476742 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:56.471821    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:56.476742 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:56.472270    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:56.477900 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:56.477644    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:56.497739 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:56.496807    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 00:57:56.576390 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:56.575212    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": context deadline exceeded"
Oct 30 00:57:56.577865 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:56.577698    6099 generic.go:356] "Generic (PLEG): container finished" podID="0519dba7-7145-4761-b18c-397c0bee13d8" containerID="ec32240736c4e6473de34d4ef494e17744d70f751789fea6ed2d3c724a0dd2d2" exitCode=0
Oct 30 00:57:56.578413 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:56.578370    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg" event={"ID":"0519dba7-7145-4761-b18c-397c0bee13d8","Type":"ContainerDied","Data":"ec32240736c4e6473de34d4ef494e17744d70f751789fea6ed2d3c724a0dd2d2"}
Oct 30 00:57:56.721069 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:56.720965    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 00:57:57.192729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.186129    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:57.295082 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.284795    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:57.295082 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.285399    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:57:57.447101 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.394553    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:57.447101 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.395818    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:57.447101 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.396581    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:57:57.601635 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.601156    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 00:57:57.606729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.602410    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:57:57.630556 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.629429    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:57.631411 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.631032    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:57:57.631411 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.631100    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:57:57.631411 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.631137    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:57:57.670787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.635874    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg"
Oct 30 00:57:57.689014 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.688946    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:57:57.881491 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.737975    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": context deadline exceeded"
Oct 30 00:57:57.881491 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.763793    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:57:57.881491 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.822818    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 00:57:57.881491 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.823140    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:57:57.937358 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.909248    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 00:57:57.937358 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.920783    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:57:57.937358 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.928938    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Oct 30 00:57:57.982906 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:57.982863    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:58.208904 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.170966    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg" event={"ID":"0519dba7-7145-4761-b18c-397c0bee13d8","Type":"ContainerDied","Data":"80141954cca8197956e6dde324596cc5916c5e6dc26015b6af06d1f8c86d2563"}
Oct 30 00:57:58.208904 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.171045    6099 scope.go:122] "RemoveContainer" containerID="ec32240736c4e6473de34d4ef494e17744d70f751789fea6ed2d3c724a0dd2d2"
Oct 30 00:57:58.208904 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.171612    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg"
Oct 30 00:57:58.286066 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.286022    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:58.344188 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.341657    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:58.399846 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.396011    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:58.416800 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.410826    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:58.557636 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.554825    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:58.571871 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.571847    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:57:58.574404 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.574367    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 00:57:58.575600 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.575577    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0519dba7-7145-4761-b18c-397c0bee13d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0519dba7-7145-4761-b18c-397c0bee13d8-plugins-config-volume\") pod \"0519dba7-7145-4761-b18c-397c0bee13d8\" (UID: \"0519dba7-7145-4761-b18c-397c0bee13d8\") "
Oct 30 00:57:58.576148 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.576128    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0519dba7-7145-4761-b18c-397c0bee13d8-kube-api-access-v477s\" (UniqueName: \"kubernetes.io/projected/0519dba7-7145-4761-b18c-397c0bee13d8-kube-api-access-v477s\") pod \"0519dba7-7145-4761-b18c-397c0bee13d8\" (UID: \"0519dba7-7145-4761-b18c-397c0bee13d8\") "
Oct 30 00:57:58.578864 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.578826    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0519dba7-7145-4761-b18c-397c0bee13d8-plugins-config-volume" pod "0519dba7-7145-4761-b18c-397c0bee13d8" (UID: "0519dba7-7145-4761-b18c-397c0bee13d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:57:58.609101 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.609068    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:58.609924 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.609887    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 00:57:58.611608 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.611583    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0519dba7-7145-4761-b18c-397c0bee13d8-kube-api-access-v477s" pod "0519dba7-7145-4761-b18c-397c0bee13d8" (UID: "0519dba7-7145-4761-b18c-397c0bee13d8"). InnerVolumeSpecName "kube-api-access-v477s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:57:58.654754 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.613127    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:57:58.654754 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.643840    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"]
Oct 30 00:57:58.654754 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.631267    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:58.654754 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.649982    6099 cpu_manager.go:414] "containerMap: removing container" podUID="0519dba7-7145-4761-b18c-397c0bee13d8" containerName="payload-processing"
Oct 30 00:57:58.654754 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.650004    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0519dba7-7145-4761-b18c-397c0bee13d8" containerName="payload-processing"
Oct 30 00:57:58.654754 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.650025    6099 cpu_manager.go:414] "containerMap: removing container" podUID="fce17e2f-7d6c-4491-b1ae-3043c7f8e87a" containerName="revoke-keys"
Oct 30 00:57:58.654754 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.650034    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce17e2f-7d6c-4491-b1ae-3043c7f8e87a" containerName="revoke-keys"
Oct 30 00:57:58.654754 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.650264    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0519dba7-7145-4761-b18c-397c0bee13d8" containerName="payload-processing"
Oct 30 00:57:58.654754 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.650281    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce17e2f-7d6c-4491-b1ae-3043c7f8e87a" containerName="revoke-keys"
Oct 30 00:57:58.714919 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.714868    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"]
Oct 30 00:57:58.717456 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.717419    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:57:58.717694 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.714888    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v477s\" (UniqueName: \"kubernetes.io/projected/0519dba7-7145-4761-b18c-397c0bee13d8-kube-api-access-v477s\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:57:58.717858 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.717844    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0519dba7-7145-4761-b18c-397c0bee13d8-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:57:58.717975 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.715306    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.723597 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.723579    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 00:57:58.726733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.726715    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 00:57:58.732365 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.732312    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:57:58.742070 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.742055    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:57:58.745061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.744987    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 00:57:58.746976 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.746899    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 00:57:58.819323 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.819294    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-envoy\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.820262 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.819565    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8lmq\" (UniqueName: \"kubernetes.io/projected/6474c172-c1ba-43d1-a965-242e6b93a3a2-kube-api-access-l8lmq\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.820918 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.820611    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-podinfo\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.821555 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.821532    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-data\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.822030 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.822011    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-credential-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.822227 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.822209    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-token\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.822412 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.822394    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6474c172-c1ba-43d1-a965-242e6b93a3a2-istiod-ca-cert\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.822536 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.822526    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-workload-certs\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.822661 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.822652    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-workload-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.923821 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.923787    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8lmq\" (UniqueName: \"kubernetes.io/projected/6474c172-c1ba-43d1-a965-242e6b93a3a2-kube-api-access-l8lmq\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.923999 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.923912    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-podinfo\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.923999 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.923938    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-data\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.923999 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.923968    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-credential-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.924174 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.924036    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-token\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.924174 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.924084    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6474c172-c1ba-43d1-a965-242e6b93a3a2-istiod-ca-cert\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.924174 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.924111    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-workload-certs\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.924368 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.924219    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-workload-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.924429 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.924411    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-data\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.924489 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.924445    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-credential-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.924546 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.924512    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-envoy\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.924546 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.924515    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-workload-certs\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.925192 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.924934    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-workload-socket\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.925461 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.925437    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6474c172-c1ba-43d1-a965-242e6b93a3a2-istiod-ca-cert\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.927349 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.927325    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-envoy\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.927445 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.927354    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-podinfo\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.948200 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.948170    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8lmq\" (UniqueName: \"kubernetes.io/projected/6474c172-c1ba-43d1-a965-242e6b93a3a2-kube-api-access-l8lmq\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:58.948358 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:58.948202    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-token\") pod \"e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") " pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:59.062819 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:59.062786    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:57:59.225730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:59.220830    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg"]
Oct 30 00:57:59.245325 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:59.245280    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6867b8-757c78757b-7krfg"]
Oct 30 00:57:59.422207 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:59.422174    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"]
Oct 30 00:57:59.422972 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:57:59.422931    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6474c172_c1ba_43d1_a965_242e6b93a3a2.slice/crio-e25508c55536b130cb5c39586f2260eb079617e7dc7bcd3233bd6068ef0d75b6 WatchSource:0}: Error finding container e25508c55536b130cb5c39586f2260eb079617e7dc7bcd3233bd6068ef0d75b6: Status 404 returned error can't find the container with id e25508c55536b130cb5c39586f2260eb079617e7dc7bcd3233bd6068ef0d75b6
Oct 30 00:57:59.426937 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:59.426895    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:59.427057 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:59.426988    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:59.427057 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:57:59.427033    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:00.220860 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:00.220819    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj" event={"ID":"6474c172-c1ba-43d1-a965-242e6b93a3a2","Type":"ContainerStarted","Data":"59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764"}
Oct 30 00:58:00.221032 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:00.220884    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj" event={"ID":"6474c172-c1ba-43d1-a965-242e6b93a3a2","Type":"ContainerStarted","Data":"e25508c55536b130cb5c39586f2260eb079617e7dc7bcd3233bd6068ef0d75b6"}
Oct 30 00:58:00.264442 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:00.264411    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0519dba7-7145-4761-b18c-397c0bee13d8" path="/var/lib/kubelet/pods/0519dba7-7145-4761-b18c-397c0bee13d8/volumes"
Oct 30 00:58:00.302421 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:00.302364    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj" podStartSLOduration=33.302345034 podStartE2EDuration="33.302345034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:00.299396559 +0000 UTC m=+1374.156304317" watchObservedRunningTime="2026-10-30 00:58:00.302345034 +0000 UTC m=+1374.159252791"
Oct 30 00:58:01.063156 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:01.063114    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:58:01.064675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:01.064649    6099 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 00:58:01.064840 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:01.064722    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj" podUID="6474c172-c1ba-43d1-a965-242e6b93a3a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 00:58:02.064213 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:02.064175    6099 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 00:58:02.064769 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:02.064251    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj" podUID="6474c172-c1ba-43d1-a965-242e6b93a3a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 00:58:03.073773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:03.073300    6099 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 00:58:03.073773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:03.073378    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj" podUID="6474c172-c1ba-43d1-a965-242e6b93a3a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 00:58:03.521075 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:03.520980    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4"]
Oct 30 00:58:03.534966 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:03.534932    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-txdb4"]
Oct 30 00:58:04.064176 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:04.064131    6099 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 00:58:04.065749 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:04.064857    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj" podUID="6474c172-c1ba-43d1-a965-242e6b93a3a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 00:58:04.309309 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:04.309270    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c744876-e4fc-4433-8bcd-7c490336f1da" path="/var/lib/kubelet/pods/7c744876-e4fc-4433-8bcd-7c490336f1da/volumes"
Oct 30 00:58:05.013368 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:05.013329    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"]
Oct 30 00:58:05.013592 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:05.013564    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv" podUID="cbccdfc4-450a-4048-9f3b-f52be1b3bb22" containerName="istio-proxy" containerID="cri-o://62aa0933c8a17805ab731007a95af3976b838e5a6623193ad67ba15c7049225b" gracePeriod=30
Oct 30 00:58:05.028467 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:05.028049    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"]
Oct 30 00:58:05.028467 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:05.028309    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs" podUID="44d70b91-1209-43ad-b90b-28cdb782b498" containerName="istio-proxy" containerID="cri-o://8635c17ad8601e737501139dbffd06f53e73a2ea7c871854580bfe31266c14cb" gracePeriod=30
Oct 30 00:58:05.035509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:05.035478    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"]
Oct 30 00:58:05.035783 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:05.035751    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd" podUID="96e60f3b-c78f-4de5-aefe-d4e8ee216a84" containerName="istio-proxy" containerID="cri-o://6cb675297151a0c203afdb61c8cb245c1899acd1801873373314aafe7d66d6e6" gracePeriod=30
Oct 30 00:58:05.098731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:05.097099    6099 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 00:58:05.098731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:05.097167    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj" podUID="6474c172-c1ba-43d1-a965-242e6b93a3a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 00:58:05.550258 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:05.550216    6099 patch_prober.go:29] interesting pod/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:58:05.550622 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:05.550280    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn" podUID="29a75432-1847-4c51-96e8-ec245d55a103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:06.068641 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:06.068604    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:58:06.069082 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:06.069056    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:58:06.069838 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:06.069819    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 00:58:06.361879 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:06.361847    6099 generic.go:356] "Generic (PLEG): container finished" podID="29a75432-1847-4c51-96e8-ec245d55a103" containerID="784d620e38d6a18ac1b3cc95e1db0dada1e1913cc211c848857531f968a98119" exitCode=0
Oct 30 00:58:06.362067 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:06.361925    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn" event={"ID":"29a75432-1847-4c51-96e8-ec245d55a103","Type":"ContainerDied","Data":"784d620e38d6a18ac1b3cc95e1db0dada1e1913cc211c848857531f968a98119"}
Oct 30 00:58:06.362694 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:06.362679    6099 scope.go:122] "RemoveContainer" containerID="784d620e38d6a18ac1b3cc95e1db0dada1e1913cc211c848857531f968a98119"
Oct 30 00:58:06.364754 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:06.364673    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:06.364846 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:06.364802    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:06.364846 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:06.364842    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:07.372762 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:07.372729    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn" event={"ID":"29a75432-1847-4c51-96e8-ec245d55a103","Type":"ContainerStarted","Data":"06566bd0e84879d316ff1d2c44e663b6a494f1d3eafae93fe0c2e840b221dcf1"}
Oct 30 00:58:07.737991 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:07.737913    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:58:07.738137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:07.738041    6099 patch_prober.go:29] interesting pod/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 00:58:07.738137 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:07.738086    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn" podUID="29a75432-1847-4c51-96e8-ec245d55a103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 00:58:08.737848 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:08.737814    6099 patch_prober.go:29] interesting pod/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 00:58:08.738240 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:08.737886    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn" podUID="29a75432-1847-4c51-96e8-ec245d55a103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 00:58:09.663465 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:09.663425    6099 scope.go:122] "RemoveContainer" containerID="efae10d0ac8aab478c9bcbbc154e600e6a4a1fa3234b1faf041a4ac7c9b986cb"
Oct 30 00:58:09.699568 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:09.699548    6099 scope.go:122] "RemoveContainer" containerID="880a519ab11f2d5a7dce880949572d9fc0dd602e7e3d45f905da359267d8d15b"
Oct 30 00:58:09.711200 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:09.711177    6099 scope.go:122] "RemoveContainer" containerID="fdca38c392537e1e0eaa574da409a58199386ebdb2868b1b72ac34c9725e2d34"
Oct 30 00:58:09.722026 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:09.722004    6099 scope.go:122] "RemoveContainer" containerID="b0e6f2894c430e195e91be8a328f22bcab64a8079b8931a960e850d6657f8169"
Oct 30 00:58:09.747787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:09.747691    6099 patch_prober.go:29] interesting pod/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 00:58:09.748370 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:09.747811    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn" podUID="29a75432-1847-4c51-96e8-ec245d55a103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 00:58:09.813863 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:09.813832    6099 scope.go:122] "RemoveContainer" containerID="c1364c264ff91e6148c4557e558f24d4bdae5ab9f84437fd4bc28fb20a733691"
Oct 30 00:58:09.841377 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:09.841355    6099 scope.go:122] "RemoveContainer" containerID="0dbe9857f45f06441cac64c8c7b8ced3ef7af8669444c290b1069f54f3a8281b"
Oct 30 00:58:09.861155 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:09.861136    6099 scope.go:122] "RemoveContainer" containerID="e43a1feb082b8cf02eb76061fc1cfb02ea1f1cb362c1851d0c8cc0a1ac801525"
Oct 30 00:58:09.903775 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:09.903517    6099 scope.go:122] "RemoveContainer" containerID="58f1a83f0686182ce1fa97415bdd49d64486fd55a0ce65a4a623407b5491a7b6"
Oct 30 00:58:09.916502 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:09.916479    6099 scope.go:122] "RemoveContainer" containerID="e085c2d08ad3d2de13e3baa117de8afd71657d923dc189a2cbabf8da6e9673e3"
Oct 30 00:58:10.426174 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.426133    6099 generic.go:356] "Generic (PLEG): container finished" podID="cbccdfc4-450a-4048-9f3b-f52be1b3bb22" containerID="62aa0933c8a17805ab731007a95af3976b838e5a6623193ad67ba15c7049225b" exitCode=0
Oct 30 00:58:10.426330 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.426279    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv" event={"ID":"cbccdfc4-450a-4048-9f3b-f52be1b3bb22","Type":"ContainerDied","Data":"62aa0933c8a17805ab731007a95af3976b838e5a6623193ad67ba15c7049225b"}
Oct 30 00:58:10.429919 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.429887    6099 generic.go:356] "Generic (PLEG): container finished" podID="96e60f3b-c78f-4de5-aefe-d4e8ee216a84" containerID="6cb675297151a0c203afdb61c8cb245c1899acd1801873373314aafe7d66d6e6" exitCode=0
Oct 30 00:58:10.430048 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.430028    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd" event={"ID":"96e60f3b-c78f-4de5-aefe-d4e8ee216a84","Type":"ContainerDied","Data":"6cb675297151a0c203afdb61c8cb245c1899acd1801873373314aafe7d66d6e6"}
Oct 30 00:58:10.436680 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.433622    6099 generic.go:356] "Generic (PLEG): container finished" podID="44d70b91-1209-43ad-b90b-28cdb782b498" containerID="8635c17ad8601e737501139dbffd06f53e73a2ea7c871854580bfe31266c14cb" exitCode=0
Oct 30 00:58:10.436680 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.433865    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs" event={"ID":"44d70b91-1209-43ad-b90b-28cdb782b498","Type":"ContainerDied","Data":"8635c17ad8601e737501139dbffd06f53e73a2ea7c871854580bfe31266c14cb"}
Oct 30 00:58:10.454781 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.454484    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:58:10.530638 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.530592    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:58:10.547667 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.547635    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-kube-api-access-xxqp2\" (UniqueName: \"kubernetes.io/projected/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-kube-api-access-xxqp2\") pod \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") "
Oct 30 00:58:10.547949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.547867    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-data\") pod \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") "
Oct 30 00:58:10.547949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.547928    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-token\" (UniqueName: \"kubernetes.io/projected/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-token\") pod \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") "
Oct 30 00:58:10.548127 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.547987    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-workload-certs\") pod \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") "
Oct 30 00:58:10.548127 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.548055    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-envoy\") pod \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") "
Oct 30 00:58:10.548219 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.548174    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-podinfo\") pod \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") "
Oct 30 00:58:10.548275 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.548247    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-credential-socket\") pod \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") "
Oct 30 00:58:10.549560 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.548363    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istiod-ca-cert\") pod \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") "
Oct 30 00:58:10.549560 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.548448    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-workload-socket\") pod \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\" (UID: \"cbccdfc4-450a-4048-9f3b-f52be1b3bb22\") "
Oct 30 00:58:10.549560 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.548560    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:58:10.549560 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.548933    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-workload-certs" pod "cbccdfc4-450a-4048-9f3b-f52be1b3bb22" (UID: "cbccdfc4-450a-4048-9f3b-f52be1b3bb22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:10.549560 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.549474    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.549921 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.549662    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-workload-socket" pod "cbccdfc4-450a-4048-9f3b-f52be1b3bb22" (UID: "cbccdfc4-450a-4048-9f3b-f52be1b3bb22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:10.549921 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.549831    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-credential-socket" pod "cbccdfc4-450a-4048-9f3b-f52be1b3bb22" (UID: "cbccdfc4-450a-4048-9f3b-f52be1b3bb22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:10.550069 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.550044    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istiod-ca-cert" pod "cbccdfc4-450a-4048-9f3b-f52be1b3bb22" (UID: "cbccdfc4-450a-4048-9f3b-f52be1b3bb22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:10.550393 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.550356    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-data" pod "cbccdfc4-450a-4048-9f3b-f52be1b3bb22" (UID: "cbccdfc4-450a-4048-9f3b-f52be1b3bb22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:10.551834 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.551804    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-token" pod "cbccdfc4-450a-4048-9f3b-f52be1b3bb22" (UID: "cbccdfc4-450a-4048-9f3b-f52be1b3bb22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:10.551953 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.551835    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-podinfo" pod "cbccdfc4-450a-4048-9f3b-f52be1b3bb22" (UID: "cbccdfc4-450a-4048-9f3b-f52be1b3bb22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:58:10.552517 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.552492    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-envoy" pod "cbccdfc4-450a-4048-9f3b-f52be1b3bb22" (UID: "cbccdfc4-450a-4048-9f3b-f52be1b3bb22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:10.552670 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.552634    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-kube-api-access-xxqp2" pod "cbccdfc4-450a-4048-9f3b-f52be1b3bb22" (UID: "cbccdfc4-450a-4048-9f3b-f52be1b3bb22"). InnerVolumeSpecName "kube-api-access-xxqp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:10.556509 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.556487    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"]
Oct 30 00:58:10.557297 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.557281    6099 cpu_manager.go:414] "containerMap: removing container" podUID="96e60f3b-c78f-4de5-aefe-d4e8ee216a84" containerName="istio-proxy"
Oct 30 00:58:10.557297 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.557298    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e60f3b-c78f-4de5-aefe-d4e8ee216a84" containerName="istio-proxy"
Oct 30 00:58:10.557441 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.557359    6099 cpu_manager.go:414] "containerMap: removing container" podUID="44d70b91-1209-43ad-b90b-28cdb782b498" containerName="istio-proxy"
Oct 30 00:58:10.557441 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.557364    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d70b91-1209-43ad-b90b-28cdb782b498" containerName="istio-proxy"
Oct 30 00:58:10.557441 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.557373    6099 cpu_manager.go:414] "containerMap: removing container" podUID="cbccdfc4-450a-4048-9f3b-f52be1b3bb22" containerName="istio-proxy"
Oct 30 00:58:10.557441 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.557378    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbccdfc4-450a-4048-9f3b-f52be1b3bb22" containerName="istio-proxy"
Oct 30 00:58:10.557628 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.557488    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d70b91-1209-43ad-b90b-28cdb782b498" containerName="istio-proxy"
Oct 30 00:58:10.557628 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.557497    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbccdfc4-450a-4048-9f3b-f52be1b3bb22" containerName="istio-proxy"
Oct 30 00:58:10.557628 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.557506    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e60f3b-c78f-4de5-aefe-d4e8ee216a84" containerName="istio-proxy"
Oct 30 00:58:10.560536 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.560521    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.612462 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.612437    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"]
Oct 30 00:58:10.633954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.632944    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"]
Oct 30 00:58:10.638650 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.638625    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"]
Oct 30 00:58:10.638827 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.638776    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.651445 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.651421    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-envoy\") pod \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") "
Oct 30 00:58:10.651571 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.651462    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d70b91-1209-43ad-b90b-28cdb782b498-istio-token\" (UniqueName: \"kubernetes.io/projected/44d70b91-1209-43ad-b90b-28cdb782b498-istio-token\") pod \"44d70b91-1209-43ad-b90b-28cdb782b498\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") "
Oct 30 00:58:10.651571 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.651516    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d70b91-1209-43ad-b90b-28cdb782b498-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d70b91-1209-43ad-b90b-28cdb782b498-istiod-ca-cert\") pod \"44d70b91-1209-43ad-b90b-28cdb782b498\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") "
Oct 30 00:58:10.651571 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.651539    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istiod-ca-cert\") pod \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") "
Oct 30 00:58:10.651768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.651580    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44d70b91-1209-43ad-b90b-28cdb782b498-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d70b91-1209-43ad-b90b-28cdb782b498-istio-podinfo\") pod \"44d70b91-1209-43ad-b90b-28cdb782b498\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") "
Oct 30 00:58:10.651768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.651628    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-istio-envoy\") pod \"44d70b91-1209-43ad-b90b-28cdb782b498\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") "
Oct 30 00:58:10.651768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.651659    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-podinfo\") pod \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") "
Oct 30 00:58:10.651768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.651756    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-kube-api-access-px97b\" (UniqueName: \"kubernetes.io/projected/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-kube-api-access-px97b\") pod \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") "
Oct 30 00:58:10.652159 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.652136    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-istio-data\") pod \"44d70b91-1209-43ad-b90b-28cdb782b498\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") "
Oct 30 00:58:10.652230 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.652193    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-credential-socket\") pod \"44d70b91-1209-43ad-b90b-28cdb782b498\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") "
Oct 30 00:58:10.652230 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.652207    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d70b91-1209-43ad-b90b-28cdb782b498-istiod-ca-cert" pod "44d70b91-1209-43ad-b90b-28cdb782b498" (UID: "44d70b91-1209-43ad-b90b-28cdb782b498"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:10.652377 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.652360    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d70b91-1209-43ad-b90b-28cdb782b498-kube-api-access-czzpj\" (UniqueName: \"kubernetes.io/projected/44d70b91-1209-43ad-b90b-28cdb782b498-kube-api-access-czzpj\") pod \"44d70b91-1209-43ad-b90b-28cdb782b498\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") "
Oct 30 00:58:10.652418 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.652405    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-workload-certs\") pod \"44d70b91-1209-43ad-b90b-28cdb782b498\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") "
Oct 30 00:58:10.652505 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.652489    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-workload-certs\") pod \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") "
Oct 30 00:58:10.652594 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.652579    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-credential-socket\") pod \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") "
Oct 30 00:58:10.652663 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.652635    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-workload-socket\") pod \"44d70b91-1209-43ad-b90b-28cdb782b498\" (UID: \"44d70b91-1209-43ad-b90b-28cdb782b498\") "
Oct 30 00:58:10.652756 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.652665    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-workload-socket\") pod \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") "
Oct 30 00:58:10.652843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.652825    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-token\" (UniqueName: \"kubernetes.io/projected/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-token\") pod \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") "
Oct 30 00:58:10.652905 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.652856    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-data\") pod \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\" (UID: \"96e60f3b-c78f-4de5-aefe-d4e8ee216a84\") "
Oct 30 00:58:10.653139 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.653122    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.653219 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.653150    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.653286 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.653270    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.653499 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.653479    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.653576 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.653525    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.653645 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.653586    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.653722 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.653650    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6wrw\" (UniqueName: \"kubernetes.io/projected/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-kube-api-access-w6wrw\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.653792 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.653760    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.653881 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.653863    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.655248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.653984    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istiod-ca-cert" pod "96e60f3b-c78f-4de5-aefe-d4e8ee216a84" (UID: "96e60f3b-c78f-4de5-aefe-d4e8ee216a84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:10.655248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.654076    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44d70b91-1209-43ad-b90b-28cdb782b498-istio-podinfo" pod "44d70b91-1209-43ad-b90b-28cdb782b498" (UID: "44d70b91-1209-43ad-b90b-28cdb782b498"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:58:10.655248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.654286    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d70b91-1209-43ad-b90b-28cdb782b498-istio-token" pod "44d70b91-1209-43ad-b90b-28cdb782b498" (UID: "44d70b91-1209-43ad-b90b-28cdb782b498"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:10.655248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.654559    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-workload-certs" pod "44d70b91-1209-43ad-b90b-28cdb782b498" (UID: "44d70b91-1209-43ad-b90b-28cdb782b498"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:10.655248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.654801    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-istio-data" pod "44d70b91-1209-43ad-b90b-28cdb782b498" (UID: "44d70b91-1209-43ad-b90b-28cdb782b498"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:10.655248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.654828    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.655248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.654851    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.655248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.654869    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.655248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.654885    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d70b91-1209-43ad-b90b-28cdb782b498-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.655248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.654897    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.655248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.654909    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.655248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.654921    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxqp2\" (UniqueName: \"kubernetes.io/projected/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-kube-api-access-xxqp2\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.655248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.654939    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.655248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.654953    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.655248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.654965    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbccdfc4-450a-4048-9f3b-f52be1b3bb22-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.655248 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.655009    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-credential-socket" pod "44d70b91-1209-43ad-b90b-28cdb782b498" (UID: "44d70b91-1209-43ad-b90b-28cdb782b498"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:10.656682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.655312    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-envoy" pod "96e60f3b-c78f-4de5-aefe-d4e8ee216a84" (UID: "96e60f3b-c78f-4de5-aefe-d4e8ee216a84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:10.656682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.655603    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-credential-socket" pod "96e60f3b-c78f-4de5-aefe-d4e8ee216a84" (UID: "96e60f3b-c78f-4de5-aefe-d4e8ee216a84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:10.656682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.655637    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-podinfo" pod "96e60f3b-c78f-4de5-aefe-d4e8ee216a84" (UID: "96e60f3b-c78f-4de5-aefe-d4e8ee216a84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:58:10.656682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.655914    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-workload-certs" pod "96e60f3b-c78f-4de5-aefe-d4e8ee216a84" (UID: "96e60f3b-c78f-4de5-aefe-d4e8ee216a84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:10.656682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.656106    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-workload-socket" pod "96e60f3b-c78f-4de5-aefe-d4e8ee216a84" (UID: "96e60f3b-c78f-4de5-aefe-d4e8ee216a84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:10.656682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.656251    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-workload-socket" pod "44d70b91-1209-43ad-b90b-28cdb782b498" (UID: "44d70b91-1209-43ad-b90b-28cdb782b498"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:10.656682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.656325    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-token" pod "96e60f3b-c78f-4de5-aefe-d4e8ee216a84" (UID: "96e60f3b-c78f-4de5-aefe-d4e8ee216a84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:10.656682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.656335    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-data" pod "96e60f3b-c78f-4de5-aefe-d4e8ee216a84" (UID: "96e60f3b-c78f-4de5-aefe-d4e8ee216a84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:10.656682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.656625    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-kube-api-access-px97b" pod "96e60f3b-c78f-4de5-aefe-d4e8ee216a84" (UID: "96e60f3b-c78f-4de5-aefe-d4e8ee216a84"). InnerVolumeSpecName "kube-api-access-px97b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:10.656682 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.656677    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-istio-envoy" pod "44d70b91-1209-43ad-b90b-28cdb782b498" (UID: "44d70b91-1209-43ad-b90b-28cdb782b498"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:10.658371 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.658346    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d70b91-1209-43ad-b90b-28cdb782b498-kube-api-access-czzpj" pod "44d70b91-1209-43ad-b90b-28cdb782b498" (UID: "44d70b91-1209-43ad-b90b-28cdb782b498"). InnerVolumeSpecName "kube-api-access-czzpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:10.728294 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.727355    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"]
Oct 30 00:58:10.733433 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.733365    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.738061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.738024    6099 patch_prober.go:29] interesting pod/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 00:58:10.738202 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.738073    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn" podUID="29a75432-1847-4c51-96e8-ec245d55a103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 00:58:10.743440 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.743387    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"]
Oct 30 00:58:10.756257 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.756223    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.756632 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.756289    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt4f9\" (UniqueName: \"kubernetes.io/projected/99e50152-55fb-4e62-a4af-0efbb35fc8df-kube-api-access-nt4f9\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.756632 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.756335    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e50152-55fb-4e62-a4af-0efbb35fc8df-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.756632 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.756467    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.756632 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.756502    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.756632 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.756552    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.756632 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.756584    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.756632 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.756631    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6wrw\" (UniqueName: \"kubernetes.io/projected/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-kube-api-access-w6wrw\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.757106 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.756671    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.757106 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.756751    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.757106 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.756783    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.757106 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.756813    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.757106 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.756888    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.757106 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757059    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.757106 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757092    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.757466 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757160    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.757466 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757185    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.757466 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757270    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.757466 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757372    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.757466 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757388    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.757466 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757401    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px97b\" (UniqueName: \"kubernetes.io/projected/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-kube-api-access-px97b\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.757466 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757415    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.757466 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757428    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.757466 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757443    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czzpj\" (UniqueName: \"kubernetes.io/projected/44d70b91-1209-43ad-b90b-28cdb782b498-kube-api-access-czzpj\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.757466 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757457    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.757466 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757471    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.758041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757487    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.758041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757501    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d70b91-1209-43ad-b90b-28cdb782b498-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.758041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757516    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.758041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757531    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.758041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757543    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.758041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757555    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e60f3b-c78f-4de5-aefe-d4e8ee216a84-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.758041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757566    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d70b91-1209-43ad-b90b-28cdb782b498-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.758041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.757578    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d70b91-1209-43ad-b90b-28cdb782b498-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:10.758430 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.758182    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.760946 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.759262    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.760946 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.759896    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.760946 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.760195    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.760946 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.760587    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.761912 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.761870    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.762001 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.761952    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.803416 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.798488    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6wrw\" (UniqueName: \"kubernetes.io/projected/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-kube-api-access-w6wrw\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.803416 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.803325    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.859497 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.859453    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75118b09-18bb-4678-b966-5df63a1720bf-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.859694 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.859510    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48cbl\" (UniqueName: \"kubernetes.io/projected/75118b09-18bb-4678-b966-5df63a1720bf-kube-api-access-48cbl\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.859694 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.859643    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.859694 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.859678    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75118b09-18bb-4678-b966-5df63a1720bf-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.859914 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.859775    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75118b09-18bb-4678-b966-5df63a1720bf-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.859914 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.859810    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.859914 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.859848    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.859914 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.859911    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.860133 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.859967    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt4f9\" (UniqueName: \"kubernetes.io/projected/99e50152-55fb-4e62-a4af-0efbb35fc8df-kube-api-access-nt4f9\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.860133 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.859998    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.860133 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.860025    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e50152-55fb-4e62-a4af-0efbb35fc8df-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.860133 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.860079    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.860358 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.860172    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.860358 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.860212    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.860358 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.860222    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.860358 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.860264    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.860358 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.860282    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.860358 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.860300    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.860358 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.860324    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.860759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.860401    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.860759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.860498    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.860882 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.860809    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e50152-55fb-4e62-a4af-0efbb35fc8df-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.861062 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.861039    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.862787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.862765    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.862887 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.862827    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.892260 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.892229    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.892900 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.892882    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt4f9\" (UniqueName: \"kubernetes.io/projected/99e50152-55fb-4e62-a4af-0efbb35fc8df-kube-api-access-nt4f9\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.925386 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.925356    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:10.954128 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.954097    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:10.962759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.962257    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.962759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.962360    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.962759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.962419    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75118b09-18bb-4678-b966-5df63a1720bf-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.962759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.962470    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48cbl\" (UniqueName: \"kubernetes.io/projected/75118b09-18bb-4678-b966-5df63a1720bf-kube-api-access-48cbl\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.962759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.962521    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.962759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.962550    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75118b09-18bb-4678-b966-5df63a1720bf-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.962759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.962555    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.962759 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.962727    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.963455 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.962834    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75118b09-18bb-4678-b966-5df63a1720bf-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.963455 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.962961    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.963455 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.963055    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.963455 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.963259    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.963455 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.963383    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75118b09-18bb-4678-b966-5df63a1720bf-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.963455 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.963399    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.992152 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.992119    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75118b09-18bb-4678-b966-5df63a1720bf-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.992511 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.992482    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.995161 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.995106    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48cbl\" (UniqueName: \"kubernetes.io/projected/75118b09-18bb-4678-b966-5df63a1720bf-kube-api-access-48cbl\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:10.997386 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:10.997349    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75118b09-18bb-4678-b966-5df63a1720bf-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:11.063768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.063404    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:11.301474 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.301447    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"]
Oct 30 00:58:11.304554 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:58:11.304520    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d0a6b5_42e2_4180_84f2_9bd6a686da0e.slice/crio-25e04858b66f1ad7e6a7ce57f87ac491a1e14089b647aadc928f4825f14c58fd WatchSource:0}: Error finding container 25e04858b66f1ad7e6a7ce57f87ac491a1e14089b647aadc928f4825f14c58fd: Status 404 returned error can't find the container with id 25e04858b66f1ad7e6a7ce57f87ac491a1e14089b647aadc928f4825f14c58fd
Oct 30 00:58:11.311191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.311148    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:11.311294 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.311243    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:11.311294 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.311288    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:11.322747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.320829    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"]
Oct 30 00:58:11.326416 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:58:11.326393    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e50152_55fb_4e62_a4af_0efbb35fc8df.slice/crio-94e5a4162750393ba1fb1e29ab4364bfe32837e2ca6f8b19019df6961fd2e032 WatchSource:0}: Error finding container 94e5a4162750393ba1fb1e29ab4364bfe32837e2ca6f8b19019df6961fd2e032: Status 404 returned error can't find the container with id 94e5a4162750393ba1fb1e29ab4364bfe32837e2ca6f8b19019df6961fd2e032
Oct 30 00:58:11.333309 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.333273    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:11.333381 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.333361    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:11.333427 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.333404    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:11.446699 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.446672    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"
Oct 30 00:58:11.446699 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.446701    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv" event={"ID":"cbccdfc4-450a-4048-9f3b-f52be1b3bb22","Type":"ContainerDied","Data":"dd6bc0b5aba47383ce0f97d405aefc3f9acb3d06ad27eb71f487d3f55ec289a6"}
Oct 30 00:58:11.446904 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.446771    6099 scope.go:122] "RemoveContainer" containerID="62aa0933c8a17805ab731007a95af3976b838e5a6623193ad67ba15c7049225b"
Oct 30 00:58:11.447935 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.447863    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"]
Oct 30 00:58:11.449034 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.449001    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd" event={"ID":"96e60f3b-c78f-4de5-aefe-d4e8ee216a84","Type":"ContainerDied","Data":"dcbe38894977efafac0c55bd2b055b0b3bb4078cf0408dc841f2999575a34e46"}
Oct 30 00:58:11.449199 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.449137    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"
Oct 30 00:58:11.452144 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.452007    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs" event={"ID":"44d70b91-1209-43ad-b90b-28cdb782b498","Type":"ContainerDied","Data":"291b072f4e5df52e257667dc7ab251a429f3a20cec61a6590184bae3eb7bc507"}
Oct 30 00:58:11.452144 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.452115    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"
Oct 30 00:58:11.461522 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.461479    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd" event={"ID":"03d0a6b5-42e2-4180-84f2-9bd6a686da0e","Type":"ContainerStarted","Data":"d3b278a654e7d3f80a3faf005e2a447d9b2bbe969694b0c33579f7bfc931a7b0"}
Oct 30 00:58:11.461731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.461695    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd" event={"ID":"03d0a6b5-42e2-4180-84f2-9bd6a686da0e","Type":"ContainerStarted","Data":"25e04858b66f1ad7e6a7ce57f87ac491a1e14089b647aadc928f4825f14c58fd"}
Oct 30 00:58:11.494666 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.494642    6099 scope.go:122] "RemoveContainer" containerID="6cb675297151a0c203afdb61c8cb245c1899acd1801873373314aafe7d66d6e6"
Oct 30 00:58:11.504286 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.504149    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:11.504286 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.504243    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:11.504451 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.504292    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:11.519739 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.519691    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv" event={"ID":"99e50152-55fb-4e62-a4af-0efbb35fc8df","Type":"ContainerStarted","Data":"91a62bc9e59c814f5393173a63c87772d25d24af64291d5ff43c81d486a29b0f"}
Oct 30 00:58:11.519985 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.519964    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv" event={"ID":"99e50152-55fb-4e62-a4af-0efbb35fc8df","Type":"ContainerStarted","Data":"94e5a4162750393ba1fb1e29ab4364bfe32837e2ca6f8b19019df6961fd2e032"}
Oct 30 00:58:11.553777 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.552386    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd" podStartSLOduration=36.55236534 podStartE2EDuration="36.55236534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:11.519868658 +0000 UTC m=+1385.376776415" watchObservedRunningTime="2026-10-30 00:58:11.55236534 +0000 UTC m=+1385.409273090"
Oct 30 00:58:11.562745 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.562678    6099 scope.go:122] "RemoveContainer" containerID="8635c17ad8601e737501139dbffd06f53e73a2ea7c871854580bfe31266c14cb"
Oct 30 00:58:11.626961 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.626933    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"]
Oct 30 00:58:11.631894 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.631863    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-6wbpd"]
Oct 30 00:58:11.718874 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.718830    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"]
Oct 30 00:58:11.739914 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.736771    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-5pjlv"]
Oct 30 00:58:11.739914 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.737499    6099 patch_prober.go:29] interesting pod/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 00:58:11.739914 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.737550    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn" podUID="29a75432-1847-4c51-96e8-ec245d55a103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 00:58:11.785829 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.776934    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv" podStartSLOduration=36.776914827 podStartE2EDuration="36.776914827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:11.762433937 +0000 UTC m=+1385.619341684" watchObservedRunningTime="2026-10-30 00:58:11.776914827 +0000 UTC m=+1385.633822582"
Oct 30 00:58:11.862558 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.862435    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"]
Oct 30 00:58:11.912789 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.912740    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-wgqzs"]
Oct 30 00:58:11.926399 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.926361    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:11.954909 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:11.954868    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:12.275070 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:12.274990    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d70b91-1209-43ad-b90b-28cdb782b498" path="/var/lib/kubelet/pods/44d70b91-1209-43ad-b90b-28cdb782b498/volumes"
Oct 30 00:58:12.276109 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:12.276086    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e60f3b-c78f-4de5-aefe-d4e8ee216a84" path="/var/lib/kubelet/pods/96e60f3b-c78f-4de5-aefe-d4e8ee216a84/volumes"
Oct 30 00:58:12.277664 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:12.277643    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbccdfc4-450a-4048-9f3b-f52be1b3bb22" path="/var/lib/kubelet/pods/cbccdfc4-450a-4048-9f3b-f52be1b3bb22/volumes"
Oct 30 00:58:12.937739 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:12.937037    6099 patch_prober.go:29] interesting pod/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 00:58:12.937739 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:12.937098    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn" podUID="29a75432-1847-4c51-96e8-ec245d55a103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 00:58:12.945432 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:12.945074    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 00:58:12.945684 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:12.945651    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd" podUID="03d0a6b5-42e2-4180-84f2-9bd6a686da0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 00:58:12.985746 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:12.964256    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 00:58:12.985746 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:12.964319    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv" podUID="99e50152-55fb-4e62-a4af-0efbb35fc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 00:58:13.100549 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:13.100501    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm" event={"ID":"75118b09-18bb-4678-b966-5df63a1720bf","Type":"ContainerStarted","Data":"8d06aa87ba2dd99a82f43b6f6f8f45a732298415973e5ec73cc89e7193721860"}
Oct 30 00:58:13.119938 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:13.119901    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm" event={"ID":"75118b09-18bb-4678-b966-5df63a1720bf","Type":"ContainerStarted","Data":"2c061ea7e15181520cc39ae0f330be3830386252b4131c992cdff7977f1d1897"}
Oct 30 00:58:13.323741 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:13.302312    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm" podStartSLOduration=38.302289375 podStartE2EDuration="38.302289375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:13.233627675 +0000 UTC m=+1387.090535432" watchObservedRunningTime="2026-10-30 00:58:13.302289375 +0000 UTC m=+1387.159197131"
Oct 30 00:58:13.862796 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:13.834509    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:13.862796 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:13.836426    6099 patch_prober.go:29] interesting pod/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 00:58:13.862796 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:13.836464    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn" podUID="29a75432-1847-4c51-96e8-ec245d55a103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 00:58:13.862796 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:13.837221    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:13.862796 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:13.854729    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 00:58:13.889731 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:13.868100    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:13.891446 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:13.891404    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:13.891905 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:13.891877    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 00:58:13.934873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:13.934825    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:13.989904 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:13.989866    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 00:58:13.990525 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:13.990483    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv" podUID="99e50152-55fb-4e62-a4af-0efbb35fc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 00:58:14.027730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.020898    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 00:58:14.027730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.020963    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd" podUID="03d0a6b5-42e2-4180-84f2-9bd6a686da0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 00:58:14.158523 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.064080    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:14.261730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.260892    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 30 00:58:14.261730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.260963    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm" podUID="75118b09-18bb-4678-b966-5df63a1720bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 30 00:58:14.305058 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.299830    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:14.531788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.507996    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:58:14.598729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.572379    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 00:58:14.646532 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.646087    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:14.752941 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.752908    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:58:14.754368 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.754311    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:58:14.759832 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.759810    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 00:58:14.856918 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.856321    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:14.856918 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.856597    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 00:58:14.857285 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.857144    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:58:14.857285 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.857206    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:58:14.863652 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.858935    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 00:58:14.864698 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.863963    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:58:14.908730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.905846    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:14.908730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.906412    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:58:14.949730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.949047    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 00:58:14.949730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.949106    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd" podUID="03d0a6b5-42e2-4180-84f2-9bd6a686da0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 00:58:14.951790 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.950311    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:58:14.951790 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.950669    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:58:14.964192 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.963414    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 00:58:14.964192 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.963461    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv" podUID="99e50152-55fb-4e62-a4af-0efbb35fc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 00:58:14.964515 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:14.964488    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 00:58:15.438573 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:15.438482    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 30 00:58:15.442266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:15.442227    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm" podUID="75118b09-18bb-4678-b966-5df63a1720bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 30 00:58:15.453126 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:15.453083    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 00:58:15.741788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:15.738230    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 00:58:15.765729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:15.747078    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 00:58:15.908356 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:15.870614    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:15.980299 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:15.980243    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 00:58:15.986230 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:15.986192    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 00:58:15.986603 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:15.986549    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd" podUID="03d0a6b5-42e2-4180-84f2-9bd6a686da0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 00:58:15.988235 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:15.988214    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 00:58:15.988444 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:15.988422    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv" podUID="99e50152-55fb-4e62-a4af-0efbb35fc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 00:58:16.076729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:16.069838    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 30 00:58:16.143737 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:16.073927    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm" podUID="75118b09-18bb-4678-b966-5df63a1720bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 30 00:58:16.312106 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:16.312055    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:16.388780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:16.367428    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 00:58:16.805732 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:58:16.795614    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:58:16.947738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:16.942894    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 00:58:16.947738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:16.942971    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd" podUID="03d0a6b5-42e2-4180-84f2-9bd6a686da0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 00:58:16.955501 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:16.955466    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 00:58:16.955681 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:16.955516    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv" podUID="99e50152-55fb-4e62-a4af-0efbb35fc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 00:58:17.078019 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:17.070017    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 30 00:58:17.078019 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:17.070076    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm" podUID="75118b09-18bb-4678-b966-5df63a1720bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 30 00:58:17.927662 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:17.927632    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 00:58:17.928313 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:17.928278    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd" podUID="03d0a6b5-42e2-4180-84f2-9bd6a686da0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 00:58:17.969991 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:17.966753    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 00:58:17.969991 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:17.966812    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv" podUID="99e50152-55fb-4e62-a4af-0efbb35fc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 00:58:18.086878 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:18.086831    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 30 00:58:18.087184 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:18.087138    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm" podUID="75118b09-18bb-4678-b966-5df63a1720bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 30 00:58:18.931617 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:18.931577    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 00:58:18.981446 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:18.954460    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 00:58:18.982883 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:18.982825    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv" podUID="99e50152-55fb-4e62-a4af-0efbb35fc8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 00:58:19.020897 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:18.981395    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd" podUID="03d0a6b5-42e2-4180-84f2-9bd6a686da0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 00:58:19.082845 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:19.082813    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 30 00:58:19.083132 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:19.083107    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm" podUID="75118b09-18bb-4678-b966-5df63a1720bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 30 00:58:19.930414 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:19.930378    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:19.930833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:19.930799    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:19.931449 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:19.931427    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:58:19.959968 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:19.959941    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:19.960308 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:19.960208    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:19.961013 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:19.960987    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:58:20.234803 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:20.233750    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:20.235192 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:20.235115    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:20.250734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:20.236692    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:58:28.817904 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:28.817867    6099 patch_prober.go:29] interesting pod/e2e-api-b-601aea-openshift-default-8559b84459-xs799 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:58:28.818438 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:28.817938    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799" podUID="90b346c4-cc85-4fd7-9578-0533e8e902a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:31.489561 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:31.489522    6099 generic.go:356] "Generic (PLEG): container finished" podID="90b346c4-cc85-4fd7-9578-0533e8e902a9" containerID="1652164599eab84070cc7219bc6df07fdc449c1c9cfa7661091ff2acc9113ac7" exitCode=0
Oct 30 00:58:31.490022 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:31.489572    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799" event={"ID":"90b346c4-cc85-4fd7-9578-0533e8e902a9","Type":"ContainerDied","Data":"1652164599eab84070cc7219bc6df07fdc449c1c9cfa7661091ff2acc9113ac7"}
Oct 30 00:58:31.490069 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:31.490052    6099 scope.go:122] "RemoveContainer" containerID="1652164599eab84070cc7219bc6df07fdc449c1c9cfa7661091ff2acc9113ac7"
Oct 30 00:58:31.492171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:31.492143    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:31.492254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:31.492207    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:31.492254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:31.492237    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:32.503717 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:32.503668    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799" event={"ID":"90b346c4-cc85-4fd7-9578-0533e8e902a9","Type":"ContainerStarted","Data":"97f29da7df242a4b2f87af0838c1102f353873dab06d5d06c31431d2559210cc"}
Oct 30 00:58:33.014677 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:33.014638    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:58:33.014986 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:33.014963    6099 patch_prober.go:29] interesting pod/e2e-api-b-601aea-openshift-default-8559b84459-xs799 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 30 00:58:33.015039 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:33.015021    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799" podUID="90b346c4-cc85-4fd7-9578-0533e8e902a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 30 00:58:34.015138 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:34.015105    6099 patch_prober.go:29] interesting pod/e2e-api-b-601aea-openshift-default-8559b84459-xs799 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 30 00:58:34.015523 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:34.015154    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799" podUID="90b346c4-cc85-4fd7-9578-0533e8e902a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 30 00:58:34.949782 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:34.949669    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"]
Oct 30 00:58:34.952560 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:34.950380    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f" podUID="4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" containerName="istio-proxy" containerID="cri-o://3c9bc3a0c17ba83770c3f71845a248977550d879d8ec56b14d14232b4ded035c" gracePeriod=30
Oct 30 00:58:34.962357 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:34.962334    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"]
Oct 30 00:58:34.962662 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:34.962602    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt" podUID="2fc4ad98-ba46-4c8d-911e-dad3e0617670" containerName="istio-proxy" containerID="cri-o://6645e41fe6276ad0221ca2df67a542faea1b83ff51be74551e39f6e04aa42722" gracePeriod=30
Oct 30 00:58:34.987730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:34.975404    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"]
Oct 30 00:58:34.987730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:34.975649    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b" podUID="a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" containerName="istio-proxy" containerID="cri-o://334158f39fa6186d7e4dbced2fde62fee9e9be8a792168c1912ac1f92a6e2272" gracePeriod=30
Oct 30 00:58:35.014585 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:35.014552    6099 patch_prober.go:29] interesting pod/e2e-api-b-601aea-openshift-default-8559b84459-xs799 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 30 00:58:35.014813 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:35.014786    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799" podUID="90b346c4-cc85-4fd7-9578-0533e8e902a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 30 00:58:36.015021 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:36.014988    6099 patch_prober.go:29] interesting pod/e2e-api-b-601aea-openshift-default-8559b84459-xs799 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 30 00:58:36.015406 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:36.015039    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799" podUID="90b346c4-cc85-4fd7-9578-0533e8e902a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 30 00:58:37.019907 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:37.014636    6099 patch_prober.go:29] interesting pod/e2e-api-b-601aea-openshift-default-8559b84459-xs799 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 30 00:58:37.019907 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:37.014698    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799" podUID="90b346c4-cc85-4fd7-9578-0533e8e902a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 30 00:58:37.539558 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:37.539521    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w"]
Oct 30 00:58:37.543500 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:37.543481    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w"
Oct 30 00:58:37.562918 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:37.562890    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w"]
Oct 30 00:58:37.565052 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:37.565024    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c213fa-596e-4df6-9966-ad1f6c56a30e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-610065-mxq8w\" (UID: \"f1c213fa-596e-4df6-9966-ad1f6c56a30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w"
Oct 30 00:58:37.568836 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:37.568805    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzwpm\" (UniqueName: \"kubernetes.io/projected/f1c213fa-596e-4df6-9966-ad1f6c56a30e-kube-api-access-vzwpm\") pod \"maas-api-revoke-keys-e2e-api-a-610065-mxq8w\" (UID: \"f1c213fa-596e-4df6-9966-ad1f6c56a30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w"
Oct 30 00:58:37.569240 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:37.569218    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c213fa-596e-4df6-9966-ad1f6c56a30e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-610065-mxq8w\" (UID: \"f1c213fa-596e-4df6-9966-ad1f6c56a30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w"
Oct 30 00:58:37.595015 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:37.594980    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:58:37.595220 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:37.595189    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:58:37.689818 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:37.677035    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzwpm\" (UniqueName: \"kubernetes.io/projected/f1c213fa-596e-4df6-9966-ad1f6c56a30e-kube-api-access-vzwpm\") pod \"maas-api-revoke-keys-e2e-api-a-610065-mxq8w\" (UID: \"f1c213fa-596e-4df6-9966-ad1f6c56a30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w"
Oct 30 00:58:37.690085 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:37.690053    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c213fa-596e-4df6-9966-ad1f6c56a30e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-610065-mxq8w\" (UID: \"f1c213fa-596e-4df6-9966-ad1f6c56a30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w"
Oct 30 00:58:37.690310 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:37.690295    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c213fa-596e-4df6-9966-ad1f6c56a30e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-610065-mxq8w\" (UID: \"f1c213fa-596e-4df6-9966-ad1f6c56a30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w"
Oct 30 00:58:37.690938 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:37.690912    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c213fa-596e-4df6-9966-ad1f6c56a30e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-610065-mxq8w\" (UID: \"f1c213fa-596e-4df6-9966-ad1f6c56a30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w"
Oct 30 00:58:37.692809 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:37.692782    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c213fa-596e-4df6-9966-ad1f6c56a30e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-610065-mxq8w\" (UID: \"f1c213fa-596e-4df6-9966-ad1f6c56a30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w"
Oct 30 00:58:37.711677 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:37.711650    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzwpm\" (UniqueName: \"kubernetes.io/projected/f1c213fa-596e-4df6-9966-ad1f6c56a30e-kube-api-access-vzwpm\") pod \"maas-api-revoke-keys-e2e-api-a-610065-mxq8w\" (UID: \"f1c213fa-596e-4df6-9966-ad1f6c56a30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w"
Oct 30 00:58:37.858089 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:37.858013    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w"
Oct 30 00:58:38.014787 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:38.014747    6099 patch_prober.go:29] interesting pod/e2e-api-b-601aea-openshift-default-8559b84459-xs799 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 30 00:58:38.014995 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:38.014809    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799" podUID="90b346c4-cc85-4fd7-9578-0533e8e902a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 30 00:58:38.188383 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:58:38.188343    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c213fa_596e_4df6_9966_ad1f6c56a30e.slice/crio-5e4d1dd950bcf019025868624aa7cb02e127db4233b1c6799a4a9610632854e2 WatchSource:0}: Error finding container 5e4d1dd950bcf019025868624aa7cb02e127db4233b1c6799a4a9610632854e2: Status 404 returned error can't find the container with id 5e4d1dd950bcf019025868624aa7cb02e127db4233b1c6799a4a9610632854e2
Oct 30 00:58:38.189066 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:38.189042    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w"]
Oct 30 00:58:38.630505 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:38.630459    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w" event={"ID":"f1c213fa-596e-4df6-9966-ad1f6c56a30e","Type":"ContainerStarted","Data":"dabf41fbd737ba48c1143200928866ccd5ea3f82d5430d2b797778c3668f4b92"}
Oct 30 00:58:38.630687 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:38.630534    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w" event={"ID":"f1c213fa-596e-4df6-9966-ad1f6c56a30e","Type":"ContainerStarted","Data":"5e4d1dd950bcf019025868624aa7cb02e127db4233b1c6799a4a9610632854e2"}
Oct 30 00:58:38.670355 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:38.670308    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w" podStartSLOduration=1.670293729 podStartE2EDuration="1.670293729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:38.669138875 +0000 UTC m=+1412.526046630" watchObservedRunningTime="2026-10-30 00:58:38.670293729 +0000 UTC m=+1412.527201483"
Oct 30 00:58:39.020401 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:39.020323    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:58:39.021222 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:39.021201    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:58:39.021879 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:39.021847    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 00:58:39.641671 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:39.641633    6099 generic.go:356] "Generic (PLEG): container finished" podID="f1c213fa-596e-4df6-9966-ad1f6c56a30e" containerID="dabf41fbd737ba48c1143200928866ccd5ea3f82d5430d2b797778c3668f4b92" exitCode=0
Oct 30 00:58:39.642099 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:39.641722    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w" event={"ID":"f1c213fa-596e-4df6-9966-ad1f6c56a30e","Type":"ContainerDied","Data":"dabf41fbd737ba48c1143200928866ccd5ea3f82d5430d2b797778c3668f4b92"}
Oct 30 00:58:40.207776 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.207746    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:58:40.240521 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.240494    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:58:40.324861 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.324833    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"]
Oct 30 00:58:40.326652 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.326628    6099 cpu_manager.go:414] "containerMap: removing container" podUID="2fc4ad98-ba46-4c8d-911e-dad3e0617670" containerName="istio-proxy"
Oct 30 00:58:40.327047 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.327032    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc4ad98-ba46-4c8d-911e-dad3e0617670" containerName="istio-proxy"
Oct 30 00:58:40.327495 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.327482    6099 cpu_manager.go:414] "containerMap: removing container" podUID="4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" containerName="istio-proxy"
Oct 30 00:58:40.327779 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.327757    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" containerName="istio-proxy"
Oct 30 00:58:40.333284 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.333263    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" containerName="istio-proxy"
Oct 30 00:58:40.337214 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.337140    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc4ad98-ba46-4c8d-911e-dad3e0617670" containerName="istio-proxy"
Oct 30 00:58:40.342918 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.342895    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"]
Oct 30 00:58:40.343046 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.343031    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.348132 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.348106    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-podinfo\") pod \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") "
Oct 30 00:58:40.348249 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.348154    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istiod-ca-cert\") pod \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") "
Oct 30 00:58:40.348249 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.348192    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-podinfo\") pod \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") "
Oct 30 00:58:40.348249 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.348230    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-workload-socket\") pod \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") "
Oct 30 00:58:40.348369 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.348296    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-envoy\") pod \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") "
Oct 30 00:58:40.348369 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.348334    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-credential-socket\") pod \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") "
Oct 30 00:58:40.348475 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.348388    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-envoy\") pod \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") "
Oct 30 00:58:40.348475 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.348446    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-workload-certs\") pod \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") "
Oct 30 00:58:40.348583 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.348498    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-kube-api-access-kpjht\" (UniqueName: \"kubernetes.io/projected/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-kube-api-access-kpjht\") pod \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") "
Oct 30 00:58:40.348583 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.348528    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istiod-ca-cert\") pod \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") "
Oct 30 00:58:40.348697 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.348656    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-workload-certs\") pod \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") "
Oct 30 00:58:40.348795 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.348691    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-workload-socket\") pod \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") "
Oct 30 00:58:40.348847 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.348796    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc4ad98-ba46-4c8d-911e-dad3e0617670-kube-api-access-5cvvw\" (UniqueName: \"kubernetes.io/projected/2fc4ad98-ba46-4c8d-911e-dad3e0617670-kube-api-access-5cvvw\") pod \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") "
Oct 30 00:58:40.348847 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.348838    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-data\") pod \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") "
Oct 30 00:58:40.349018 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.348996    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-token\" (UniqueName: \"kubernetes.io/projected/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-token\") pod \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\" (UID: \"2fc4ad98-ba46-4c8d-911e-dad3e0617670\") "
Oct 30 00:58:40.349143 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.349028    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-data\") pod \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") "
Oct 30 00:58:40.349209 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.349185    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-credential-socket\") pod \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") "
Oct 30 00:58:40.349263 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.349251    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-token\" (UniqueName: \"kubernetes.io/projected/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-token\") pod \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\" (UID: \"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391\") "
Oct 30 00:58:40.351799 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.351221    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-data" pod "4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" (UID: "4bb3ac3a-8a32-4a49-98c3-15f3cee9f391"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:40.351799 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.351446    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-data" pod "2fc4ad98-ba46-4c8d-911e-dad3e0617670" (UID: "2fc4ad98-ba46-4c8d-911e-dad3e0617670"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:40.351799 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.351662    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-workload-socket" pod "2fc4ad98-ba46-4c8d-911e-dad3e0617670" (UID: "2fc4ad98-ba46-4c8d-911e-dad3e0617670"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:40.351799 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.351744    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-credential-socket" pod "2fc4ad98-ba46-4c8d-911e-dad3e0617670" (UID: "2fc4ad98-ba46-4c8d-911e-dad3e0617670"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:40.352896 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.352681    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc4ad98-ba46-4c8d-911e-dad3e0617670-kube-api-access-5cvvw" pod "2fc4ad98-ba46-4c8d-911e-dad3e0617670" (UID: "2fc4ad98-ba46-4c8d-911e-dad3e0617670"). InnerVolumeSpecName "kube-api-access-5cvvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:40.353346 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.353131    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istiod-ca-cert" pod "2fc4ad98-ba46-4c8d-911e-dad3e0617670" (UID: "2fc4ad98-ba46-4c8d-911e-dad3e0617670"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:40.353673 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.353488    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-workload-certs" pod "4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" (UID: "4bb3ac3a-8a32-4a49-98c3-15f3cee9f391"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:40.354221 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.353818    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-workload-socket" pod "4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" (UID: "4bb3ac3a-8a32-4a49-98c3-15f3cee9f391"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:40.354360 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.354311    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.354360 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.354336    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.354486 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.354348    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.354486 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.354463    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.354486 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.354474    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.354634 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.354486    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cvvw\" (UniqueName: \"kubernetes.io/projected/2fc4ad98-ba46-4c8d-911e-dad3e0617670-kube-api-access-5cvvw\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.354634 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.354499    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.354634 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.354511    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.355145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.354888    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istiod-ca-cert" pod "4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" (UID: "4bb3ac3a-8a32-4a49-98c3-15f3cee9f391"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:40.355145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.355005    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-credential-socket" pod "4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" (UID: "4bb3ac3a-8a32-4a49-98c3-15f3cee9f391"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:40.358814 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.358503    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-kube-api-access-kpjht" pod "4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" (UID: "4bb3ac3a-8a32-4a49-98c3-15f3cee9f391"). InnerVolumeSpecName "kube-api-access-kpjht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:40.359207 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.358078    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:58:40.360737 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.360158    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-workload-certs" pod "2fc4ad98-ba46-4c8d-911e-dad3e0617670" (UID: "2fc4ad98-ba46-4c8d-911e-dad3e0617670"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:40.360737 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.360358    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"]
Oct 30 00:58:40.360737 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.360687    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-envoy" pod "4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" (UID: "4bb3ac3a-8a32-4a49-98c3-15f3cee9f391"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:40.361177 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.361153    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-podinfo" pod "2fc4ad98-ba46-4c8d-911e-dad3e0617670" (UID: "2fc4ad98-ba46-4c8d-911e-dad3e0617670"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:58:40.361286 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.361267    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-token" pod "2fc4ad98-ba46-4c8d-911e-dad3e0617670" (UID: "2fc4ad98-ba46-4c8d-911e-dad3e0617670"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:40.361356 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.361329    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-podinfo" pod "4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" (UID: "4bb3ac3a-8a32-4a49-98c3-15f3cee9f391"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:58:40.361671 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.361606    6099 cpu_manager.go:414] "containerMap: removing container" podUID="a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" containerName="istio-proxy"
Oct 30 00:58:40.361671 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.361631    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" containerName="istio-proxy"
Oct 30 00:58:40.370810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.362459    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" containerName="istio-proxy"
Oct 30 00:58:40.370810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.363307    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-token" pod "4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" (UID: "4bb3ac3a-8a32-4a49-98c3-15f3cee9f391"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:40.370810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.366089    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.370810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.370375    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"]
Oct 30 00:58:40.372160 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.372137    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-envoy" pod "2fc4ad98-ba46-4c8d-911e-dad3e0617670" (UID: "2fc4ad98-ba46-4c8d-911e-dad3e0617670"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:40.451365 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.451333    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"]
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.456750    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.457165    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-data\") pod \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") "
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.457329    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-workload-certs\") pod \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") "
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.457478    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-data" pod "a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" (UID: "a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.457690    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istiod-ca-cert\") pod \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") "
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.457778    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-envoy\") pod \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") "
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.457786    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-workload-certs" pod "a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" (UID: "a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.457892    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-workload-socket\") pod \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") "
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.457976    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-kube-api-access-wbs6q\" (UniqueName: \"kubernetes.io/projected/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-kube-api-access-wbs6q\") pod \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") "
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.458026    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-credential-socket\") pod \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") "
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.458113    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-token\" (UniqueName: \"kubernetes.io/projected/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-token\") pod \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") "
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.458210    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-podinfo\") pod \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\" (UID: \"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8\") "
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.458619    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.458676    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.458721    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.458773    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.458816    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n67wh\" (UniqueName: \"kubernetes.io/projected/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-kube-api-access-n67wh\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.458850    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.458885    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.458918    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.458974    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istiod-ca-cert" pod "a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" (UID: "a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459086    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459129    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459184    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459224    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459322    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459441    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459502    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459533    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c030a78c-6e98-43a0-927b-a4caee58d9fa-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459626    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czm6t\" (UniqueName: \"kubernetes.io/projected/c030a78c-6e98-43a0-927b-a4caee58d9fa-kube-api-access-czm6t\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459646    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459754    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459770    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459784    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459798    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459810    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459821    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459829    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459837    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459845    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc4ad98-ba46-4c8d-911e-dad3e0617670-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459853    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpjht\" (UniqueName: \"kubernetes.io/projected/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391-kube-api-access-kpjht\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459861    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459872    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.459885    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc4ad98-ba46-4c8d-911e-dad3e0617670-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.460213    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-workload-socket" pod "a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" (UID: "a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:40.466758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.460999    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-credential-socket" pod "a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" (UID: "a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:40.471036 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.470985    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-envoy" pod "a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" (UID: "a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:58:40.487459 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.487220    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-podinfo" pod "a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" (UID: "a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:58:40.490386 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.489175    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-token" pod "a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" (UID: "a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:40.490386 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.489264    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-kube-api-access-wbs6q" pod "a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" (UID: "a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8"). InnerVolumeSpecName "kube-api-access-wbs6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:40.502770 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.502699    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"]
Oct 30 00:58:40.563400 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.563336    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdz7b\" (UniqueName: \"kubernetes.io/projected/1638dd27-e6e3-49e0-83cd-95e91893aa1c-kube-api-access-wdz7b\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.563758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.563728    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c030a78c-6e98-43a0-927b-a4caee58d9fa-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.564048 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564023    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czm6t\" (UniqueName: \"kubernetes.io/projected/c030a78c-6e98-43a0-927b-a4caee58d9fa-kube-api-access-czm6t\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.564181 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564062    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.564181 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564098    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.564181 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564129    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.564181 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564168    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.564415 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564201    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.564415 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564257    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.564415 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564294    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n67wh\" (UniqueName: \"kubernetes.io/projected/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-kube-api-access-n67wh\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.564415 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564333    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.564415 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564364    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.564415 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564398    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.564829 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564477    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.564829 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564534    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.564829 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564575    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.564829 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564599    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.564829 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564665    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.564829 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564729    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.564829 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564763    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.564829 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564792    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.564829 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564820    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.565385 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.564925    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.565385 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.565014    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.567093 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.565577    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.567093 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.566407    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c030a78c-6e98-43a0-927b-a4caee58d9fa-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.567274 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.565139    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.567405 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.567385    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.567487 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.567454    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.567593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.567578    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.567687 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.567672    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbs6q\" (UniqueName: \"kubernetes.io/projected/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-kube-api-access-wbs6q\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.567817 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.567692    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.567817 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.567720    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.567817 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.567734    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.567817 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.567750    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:40.568037 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.568020    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.568238 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.568219    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.568457 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.568437    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.568768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.568741    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.570683 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.569490    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.570683 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.569744    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.570683 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.569866    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.570683 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.570198    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.570683 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.570639    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.572619 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.572596    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.572619 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.572606    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.573412 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.573395    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.589049 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.589026    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czm6t\" (UniqueName: \"kubernetes.io/projected/c030a78c-6e98-43a0-927b-a4caee58d9fa-kube-api-access-czm6t\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.591274 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.591254    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.591500 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.591480    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.591791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.591772    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n67wh\" (UniqueName: \"kubernetes.io/projected/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-kube-api-access-n67wh\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.654422 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.654387    6099 generic.go:356] "Generic (PLEG): container finished" podID="2fc4ad98-ba46-4c8d-911e-dad3e0617670" containerID="6645e41fe6276ad0221ca2df67a542faea1b83ff51be74551e39f6e04aa42722" exitCode=0
Oct 30 00:58:40.655055 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.654455    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"
Oct 30 00:58:40.655055 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.654492    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt" event={"ID":"2fc4ad98-ba46-4c8d-911e-dad3e0617670","Type":"ContainerDied","Data":"6645e41fe6276ad0221ca2df67a542faea1b83ff51be74551e39f6e04aa42722"}
Oct 30 00:58:40.655055 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.654541    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt" event={"ID":"2fc4ad98-ba46-4c8d-911e-dad3e0617670","Type":"ContainerDied","Data":"2177855259df7dbb1103148f370112f1c25f26e129f6572c7e43c94732c09052"}
Oct 30 00:58:40.655055 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.654573    6099 scope.go:122] "RemoveContainer" containerID="6645e41fe6276ad0221ca2df67a542faea1b83ff51be74551e39f6e04aa42722"
Oct 30 00:58:40.664488 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.664456    6099 generic.go:356] "Generic (PLEG): container finished" podID="4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" containerID="3c9bc3a0c17ba83770c3f71845a248977550d879d8ec56b14d14232b4ded035c" exitCode=0
Oct 30 00:58:40.664943 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.664820    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"
Oct 30 00:58:40.664943 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.664849    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f" event={"ID":"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391","Type":"ContainerDied","Data":"3c9bc3a0c17ba83770c3f71845a248977550d879d8ec56b14d14232b4ded035c"}
Oct 30 00:58:40.664943 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.664909    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f" event={"ID":"4bb3ac3a-8a32-4a49-98c3-15f3cee9f391","Type":"ContainerDied","Data":"b2c43613a6482b81497b8337e44b5087f88e74f9fa7df9c8e311a1c52c1c85da"}
Oct 30 00:58:40.668078 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.668016    6099 generic.go:356] "Generic (PLEG): container finished" podID="a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" containerID="334158f39fa6186d7e4dbced2fde62fee9e9be8a792168c1912ac1f92a6e2272" exitCode=0
Oct 30 00:58:40.668695 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.668075    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"
Oct 30 00:58:40.668695 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.668100    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b" event={"ID":"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8","Type":"ContainerDied","Data":"334158f39fa6186d7e4dbced2fde62fee9e9be8a792168c1912ac1f92a6e2272"}
Oct 30 00:58:40.668695 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.668162    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b" event={"ID":"a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8","Type":"ContainerDied","Data":"b989c27ad2c8ca45bb3874888aae6f3aa007cdf2eed0ae42123560f50510f1a5"}
Oct 30 00:58:40.670861 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.670835    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.671457 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.671334    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-credential-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.671693 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.671671    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.672226 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.672100    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-data\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.672355 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.672174    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.672516 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.672499    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.672994 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.672953    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istiod-ca-cert\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.673081 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.672909    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-workload-certs\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.673208 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.673187    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.673782 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.673760    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.674394 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.674225    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-workload-socket\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.674923 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.674739    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:40.675045 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.675026    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.675245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.675226    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.675658 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.675636    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdz7b\" (UniqueName: \"kubernetes.io/projected/1638dd27-e6e3-49e0-83cd-95e91893aa1c-kube-api-access-wdz7b\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.676414 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.676390    6099 scope.go:122] "RemoveContainer" containerID="6645e41fe6276ad0221ca2df67a542faea1b83ff51be74551e39f6e04aa42722"
Oct 30 00:58:40.676414 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.676404    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-podinfo\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.676814 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:58:40.676762    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6645e41fe6276ad0221ca2df67a542faea1b83ff51be74551e39f6e04aa42722\": container with ID starting with 6645e41fe6276ad0221ca2df67a542faea1b83ff51be74551e39f6e04aa42722 not found: ID does not exist" containerID="6645e41fe6276ad0221ca2df67a542faea1b83ff51be74551e39f6e04aa42722"
Oct 30 00:58:40.676814 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.676804    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6645e41fe6276ad0221ca2df67a542faea1b83ff51be74551e39f6e04aa42722"} err="failed to get container status \"6645e41fe6276ad0221ca2df67a542faea1b83ff51be74551e39f6e04aa42722\": rpc error: code = NotFound desc = could not find container \"6645e41fe6276ad0221ca2df67a542faea1b83ff51be74551e39f6e04aa42722\": container with ID starting with 6645e41fe6276ad0221ca2df67a542faea1b83ff51be74551e39f6e04aa42722 not found: ID does not exist"
Oct 30 00:58:40.676994 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.676827    6099 scope.go:122] "RemoveContainer" containerID="3c9bc3a0c17ba83770c3f71845a248977550d879d8ec56b14d14232b4ded035c"
Oct 30 00:58:40.678598 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.678475    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-envoy\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.695308 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.695285    6099 scope.go:122] "RemoveContainer" containerID="3c9bc3a0c17ba83770c3f71845a248977550d879d8ec56b14d14232b4ded035c"
Oct 30 00:58:40.696589 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.696125    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:40.697474 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:58:40.697446    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c9bc3a0c17ba83770c3f71845a248977550d879d8ec56b14d14232b4ded035c\": container with ID starting with 3c9bc3a0c17ba83770c3f71845a248977550d879d8ec56b14d14232b4ded035c not found: ID does not exist" containerID="3c9bc3a0c17ba83770c3f71845a248977550d879d8ec56b14d14232b4ded035c"
Oct 30 00:58:40.697674 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.697486    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c9bc3a0c17ba83770c3f71845a248977550d879d8ec56b14d14232b4ded035c"} err="failed to get container status \"3c9bc3a0c17ba83770c3f71845a248977550d879d8ec56b14d14232b4ded035c\": rpc error: code = NotFound desc = could not find container \"3c9bc3a0c17ba83770c3f71845a248977550d879d8ec56b14d14232b4ded035c\": container with ID starting with 3c9bc3a0c17ba83770c3f71845a248977550d879d8ec56b14d14232b4ded035c not found: ID does not exist"
Oct 30 00:58:40.697674 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.697511    6099 scope.go:122] "RemoveContainer" containerID="334158f39fa6186d7e4dbced2fde62fee9e9be8a792168c1912ac1f92a6e2272"
Oct 30 00:58:40.700671 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.700619    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdz7b\" (UniqueName: \"kubernetes.io/projected/1638dd27-e6e3-49e0-83cd-95e91893aa1c-kube-api-access-wdz7b\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.701203 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.701182    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-token\") pod \"e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") " pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.734367 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.734332    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"]
Oct 30 00:58:40.741013 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.740938    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-cc8pt"]
Oct 30 00:58:40.756122 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.753888    6099 scope.go:122] "RemoveContainer" containerID="334158f39fa6186d7e4dbced2fde62fee9e9be8a792168c1912ac1f92a6e2272"
Oct 30 00:58:40.768277 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:58:40.768151    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334158f39fa6186d7e4dbced2fde62fee9e9be8a792168c1912ac1f92a6e2272\": container with ID starting with 334158f39fa6186d7e4dbced2fde62fee9e9be8a792168c1912ac1f92a6e2272 not found: ID does not exist" containerID="334158f39fa6186d7e4dbced2fde62fee9e9be8a792168c1912ac1f92a6e2272"
Oct 30 00:58:40.768277 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.768190    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334158f39fa6186d7e4dbced2fde62fee9e9be8a792168c1912ac1f92a6e2272"} err="failed to get container status \"334158f39fa6186d7e4dbced2fde62fee9e9be8a792168c1912ac1f92a6e2272\": rpc error: code = NotFound desc = could not find container \"334158f39fa6186d7e4dbced2fde62fee9e9be8a792168c1912ac1f92a6e2272\": container with ID starting with 334158f39fa6186d7e4dbced2fde62fee9e9be8a792168c1912ac1f92a6e2272 not found: ID does not exist"
Oct 30 00:58:40.794812 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.794787    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"]
Oct 30 00:58:40.806452 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.806418    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xkg8f"]
Oct 30 00:58:40.808402 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.808373    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:40.863796 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.863762    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"]
Oct 30 00:58:40.869559 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.869535    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-tmb6b"]
Oct 30 00:58:40.942131 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:40.942101    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w"
Oct 30 00:58:41.091312 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.085442    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1c213fa-596e-4df6-9966-ad1f6c56a30e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c213fa-596e-4df6-9966-ad1f6c56a30e-maas-api-openshift-ca\") pod \"f1c213fa-596e-4df6-9966-ad1f6c56a30e\" (UID: \"f1c213fa-596e-4df6-9966-ad1f6c56a30e\") "
Oct 30 00:58:41.091312 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.085594    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c213fa-596e-4df6-9966-ad1f6c56a30e-kube-api-access-vzwpm\" (UniqueName: \"kubernetes.io/projected/f1c213fa-596e-4df6-9966-ad1f6c56a30e-kube-api-access-vzwpm\") pod \"f1c213fa-596e-4df6-9966-ad1f6c56a30e\" (UID: \"f1c213fa-596e-4df6-9966-ad1f6c56a30e\") "
Oct 30 00:58:41.091312 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.085823    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1c213fa-596e-4df6-9966-ad1f6c56a30e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c213fa-596e-4df6-9966-ad1f6c56a30e-maas-api-xks-ca\") pod \"f1c213fa-596e-4df6-9966-ad1f6c56a30e\" (UID: \"f1c213fa-596e-4df6-9966-ad1f6c56a30e\") "
Oct 30 00:58:41.111128 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.110835    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c213fa-596e-4df6-9966-ad1f6c56a30e-maas-api-openshift-ca" pod "f1c213fa-596e-4df6-9966-ad1f6c56a30e" (UID: "f1c213fa-596e-4df6-9966-ad1f6c56a30e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:41.120764 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.119214    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1c213fa-596e-4df6-9966-ad1f6c56a30e-maas-api-xks-ca" pod "f1c213fa-596e-4df6-9966-ad1f6c56a30e" (UID: "f1c213fa-596e-4df6-9966-ad1f6c56a30e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:41.136151 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.136089    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c213fa-596e-4df6-9966-ad1f6c56a30e-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:41.136151 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.136136    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c213fa-596e-4df6-9966-ad1f6c56a30e-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:41.173901 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.173848    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:41.174189 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.174153    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:41.174313 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.174234    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:41.193082 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.192755    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"]
Oct 30 00:58:41.193082 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.193026    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c213fa-596e-4df6-9966-ad1f6c56a30e-kube-api-access-vzwpm" pod "f1c213fa-596e-4df6-9966-ad1f6c56a30e" (UID: "f1c213fa-596e-4df6-9966-ad1f6c56a30e"). InnerVolumeSpecName "kube-api-access-vzwpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:41.230883 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.230824    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"]
Oct 30 00:58:41.238003 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.237969    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzwpm\" (UniqueName: \"kubernetes.io/projected/f1c213fa-596e-4df6-9966-ad1f6c56a30e-kube-api-access-vzwpm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:41.239017 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.238990    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"]
Oct 30 00:58:41.242534 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:58:41.242505    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc030a78c_6e98_43a0_927b_a4caee58d9fa.slice/crio-e07bd4d9bc07ab136ec083fb3cf03eb3232b2cfe9ce4c2544bab0318cb5d3916 WatchSource:0}: Error finding container e07bd4d9bc07ab136ec083fb3cf03eb3232b2cfe9ce4c2544bab0318cb5d3916: Status 404 returned error can't find the container with id e07bd4d9bc07ab136ec083fb3cf03eb3232b2cfe9ce4c2544bab0318cb5d3916
Oct 30 00:58:41.250007 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.249906    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:41.250007 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.250000    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:41.250850 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.250045    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:41.252960 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.252368    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:41.252960 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.252460    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:41.252960 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.252510    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:41.682985 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.682945    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp" event={"ID":"cd226360-69bc-45d8-95f6-9e2d00e9e4ce","Type":"ContainerStarted","Data":"3568018eeae415a06f67b2380ae899c9d32dbedc4a95fc1dd9272b44dc12cbe8"}
Oct 30 00:58:41.683483 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.683001    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp" event={"ID":"cd226360-69bc-45d8-95f6-9e2d00e9e4ce","Type":"ContainerStarted","Data":"db5b381a165cc918caf2862ec3966fa091882828b05a7af5201f2e009e95913a"}
Oct 30 00:58:41.699893 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.699848    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:41.718932 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.718894    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w"
Oct 30 00:58:41.719673 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.718907    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w" event={"ID":"f1c213fa-596e-4df6-9966-ad1f6c56a30e","Type":"ContainerDied","Data":"5e4d1dd950bcf019025868624aa7cb02e127db4233b1c6799a4a9610632854e2"}
Oct 30 00:58:41.719673 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.719642    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e4d1dd950bcf019025868624aa7cb02e127db4233b1c6799a4a9610632854e2"
Oct 30 00:58:41.723246 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.722296    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5" event={"ID":"c030a78c-6e98-43a0-927b-a4caee58d9fa","Type":"ContainerStarted","Data":"344c718722a9b7d9c4ba28ef84052530801ecec077961775300540c5df0d73e1"}
Oct 30 00:58:41.723246 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.722348    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5" event={"ID":"c030a78c-6e98-43a0-927b-a4caee58d9fa","Type":"ContainerStarted","Data":"e07bd4d9bc07ab136ec083fb3cf03eb3232b2cfe9ce4c2544bab0318cb5d3916"}
Oct 30 00:58:41.726527 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.725768    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj" event={"ID":"1638dd27-e6e3-49e0-83cd-95e91893aa1c","Type":"ContainerStarted","Data":"2e496e88c9af041b98e19ab627d12407f0d703d70850ab109fdf8c69a13b63d8"}
Oct 30 00:58:41.726527 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.725821    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj" event={"ID":"1638dd27-e6e3-49e0-83cd-95e91893aa1c","Type":"ContainerStarted","Data":"da365b0300b77635c1db62b079e45eb4b8e2bb5b2f965e7a2f63658d45ee0cf4"}
Oct 30 00:58:41.749363 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.734230    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp" podStartSLOduration=6.734212192 podStartE2EDuration="6.734212192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:41.730680961 +0000 UTC m=+1415.587588719" watchObservedRunningTime="2026-10-30 00:58:41.734212192 +0000 UTC m=+1415.591119948"
Oct 30 00:58:41.790627 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.790566    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj" podStartSLOduration=6.79053703 podStartE2EDuration="6.79053703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:41.78391755 +0000 UTC m=+1415.640825307" watchObservedRunningTime="2026-10-30 00:58:41.79053703 +0000 UTC m=+1415.647444785"
Oct 30 00:58:41.809058 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.809029    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:41.834327 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:41.834264    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5" podStartSLOduration=6.834245087 podStartE2EDuration="6.834245087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:41.828484319 +0000 UTC m=+1415.685392074" watchObservedRunningTime="2026-10-30 00:58:41.834245087 +0000 UTC m=+1415.691152842"
Oct 30 00:58:42.399729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:42.350144    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fc4ad98-ba46-4c8d-911e-dad3e0617670" path="/var/lib/kubelet/pods/2fc4ad98-ba46-4c8d-911e-dad3e0617670/volumes"
Oct 30 00:58:42.408728 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:42.401486    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb3ac3a-8a32-4a49-98c3-15f3cee9f391" path="/var/lib/kubelet/pods/4bb3ac3a-8a32-4a49-98c3-15f3cee9f391/volumes"
Oct 30 00:58:42.421726 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:42.417724    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8" path="/var/lib/kubelet/pods/a4f0f0a1-18c0-4b97-b24e-ce5289a0cce8/volumes"
Oct 30 00:58:42.739785 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:42.728795    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:42.830732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:42.824971    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 00:58:42.830732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:42.825044    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp" podUID="cd226360-69bc-45d8-95f6-9e2d00e9e4ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 00:58:42.830732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:42.827279    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 00:58:42.830732 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:42.827325    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj" podUID="1638dd27-e6e3-49e0-83cd-95e91893aa1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 00:58:42.964348 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:42.955993    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 30 00:58:42.964348 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:42.956053    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5" podUID="c030a78c-6e98-43a0-927b-a4caee58d9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 30 00:58:43.580734 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:43.574089    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt"]
Oct 30 00:58:43.655868 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:43.653504    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:43.677139 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:43.677099    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 30 00:58:43.677432 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:43.677398    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5" podUID="c030a78c-6e98-43a0-927b-a4caee58d9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 30 00:58:43.697740 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:43.697580    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 00:58:43.697740 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:43.697642    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp" podUID="cd226360-69bc-45d8-95f6-9e2d00e9e4ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 00:58:43.717881 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:43.715369    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6867b8-756jt"]
Oct 30 00:58:43.809729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:43.806403    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:58:43.809729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:43.807248    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 00:58:43.826760 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:43.811433    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 00:58:43.827053 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:43.827018    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj" podUID="1638dd27-e6e3-49e0-83cd-95e91893aa1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 00:58:44.033532 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:44.012794    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:44.264238 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:44.264204    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fce17e2f-7d6c-4491-b1ae-3043c7f8e87a" path="/var/lib/kubelet/pods/fce17e2f-7d6c-4491-b1ae-3043c7f8e87a/volumes"
Oct 30 00:58:44.689167 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:44.689132    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 30 00:58:44.689423 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:44.689394    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5" podUID="c030a78c-6e98-43a0-927b-a4caee58d9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 30 00:58:44.743513 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:44.743474    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 00:58:44.743828 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:44.743796    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp" podUID="cd226360-69bc-45d8-95f6-9e2d00e9e4ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 00:58:44.814748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:44.811329    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 00:58:44.814748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:44.811392    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj" podUID="1638dd27-e6e3-49e0-83cd-95e91893aa1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 00:58:45.718440 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:45.718400    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 00:58:45.718779 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:45.718744    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp" podUID="cd226360-69bc-45d8-95f6-9e2d00e9e4ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 00:58:45.719364 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:45.719340    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 30 00:58:45.719536 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:45.719517    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5" podUID="c030a78c-6e98-43a0-927b-a4caee58d9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 30 00:58:45.830868 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:45.830825    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 00:58:45.831477 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:45.831442    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj" podUID="1638dd27-e6e3-49e0-83cd-95e91893aa1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 00:58:46.688151 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:46.688113    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 30 00:58:46.688437 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:46.688407    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5" podUID="c030a78c-6e98-43a0-927b-a4caee58d9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 30 00:58:46.699085 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:46.699053    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 00:58:46.699315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:46.699289    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp" podUID="cd226360-69bc-45d8-95f6-9e2d00e9e4ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 00:58:46.724784 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:58:46.724746    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:58:46.812410 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:46.812368    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 00:58:46.813193 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:46.813162    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj" podUID="1638dd27-e6e3-49e0-83cd-95e91893aa1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 00:58:47.688730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:47.688555    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 30 00:58:47.688730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:47.688626    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5" podUID="c030a78c-6e98-43a0-927b-a4caee58d9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 30 00:58:47.698319 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:47.698289    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 00:58:47.698595 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:47.698555    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp" podUID="cd226360-69bc-45d8-95f6-9e2d00e9e4ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 00:58:47.809145 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:47.809106    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 00:58:47.809402 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:47.809373    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj" podUID="1638dd27-e6e3-49e0-83cd-95e91893aa1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 00:58:48.796537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:48.796488    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:48.798655 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:48.798632    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:48.803192 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:48.803173    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:58:48.826725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:48.823104    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:48.826725 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:48.825203    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:48.827463 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:48.827366    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:58:48.841689 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:48.841655    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:48.842095 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:48.842061    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:48.843422 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:48.843392    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:58:49.819784 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:49.819751    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8"]
Oct 30 00:58:49.820615 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:49.820598    6099 cpu_manager.go:414] "containerMap: removing container" podUID="f1c213fa-596e-4df6-9966-ad1f6c56a30e" containerName="revoke-keys"
Oct 30 00:58:49.820655 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:49.820616    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c213fa-596e-4df6-9966-ad1f6c56a30e" containerName="revoke-keys"
Oct 30 00:58:49.820817 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:49.820807    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c213fa-596e-4df6-9966-ad1f6c56a30e" containerName="revoke-keys"
Oct 30 00:58:49.824364 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:49.824343    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8"
Oct 30 00:58:49.844681 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:49.844655    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8"]
Oct 30 00:58:49.864512 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:49.851539    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:58:49.864512 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:49.851604    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:58:49.908719 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:49.908672    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7584e7cf-bf67-4082-adb9-415425b02f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-416f373e-vxlm8\" (UID: \"7584e7cf-bf67-4082-adb9-415425b02f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8"
Oct 30 00:58:49.908891 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:49.908790    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7584e7cf-bf67-4082-adb9-415425b02f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-416f373e-vxlm8\" (UID: \"7584e7cf-bf67-4082-adb9-415425b02f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8"
Oct 30 00:58:49.909023 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:49.908906    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9qhh\" (UniqueName: \"kubernetes.io/projected/7584e7cf-bf67-4082-adb9-415425b02f9d-kube-api-access-b9qhh\") pod \"maas-api-revoke-sub-416f373e-vxlm8\" (UID: \"7584e7cf-bf67-4082-adb9-415425b02f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8"
Oct 30 00:58:50.010901 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:50.010869    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7584e7cf-bf67-4082-adb9-415425b02f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-416f373e-vxlm8\" (UID: \"7584e7cf-bf67-4082-adb9-415425b02f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8"
Oct 30 00:58:50.011070 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:50.010922    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7584e7cf-bf67-4082-adb9-415425b02f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-416f373e-vxlm8\" (UID: \"7584e7cf-bf67-4082-adb9-415425b02f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8"
Oct 30 00:58:50.011070 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:50.010999    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9qhh\" (UniqueName: \"kubernetes.io/projected/7584e7cf-bf67-4082-adb9-415425b02f9d-kube-api-access-b9qhh\") pod \"maas-api-revoke-sub-416f373e-vxlm8\" (UID: \"7584e7cf-bf67-4082-adb9-415425b02f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8"
Oct 30 00:58:50.011730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:50.011689    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7584e7cf-bf67-4082-adb9-415425b02f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-416f373e-vxlm8\" (UID: \"7584e7cf-bf67-4082-adb9-415425b02f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8"
Oct 30 00:58:50.013273 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:50.013251    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7584e7cf-bf67-4082-adb9-415425b02f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-416f373e-vxlm8\" (UID: \"7584e7cf-bf67-4082-adb9-415425b02f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8"
Oct 30 00:58:50.034875 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:50.034843    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9qhh\" (UniqueName: \"kubernetes.io/projected/7584e7cf-bf67-4082-adb9-415425b02f9d-kube-api-access-b9qhh\") pod \"maas-api-revoke-sub-416f373e-vxlm8\" (UID: \"7584e7cf-bf67-4082-adb9-415425b02f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8"
Oct 30 00:58:50.146857 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:50.146681    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8"
Oct 30 00:58:50.462980 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:58:50.462946    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7584e7cf_bf67_4082_adb9_415425b02f9d.slice/crio-89de15d4e97898f119e105f0cda84642e2b0ab2314004a6eae8f2603e912a921 WatchSource:0}: Error finding container 89de15d4e97898f119e105f0cda84642e2b0ab2314004a6eae8f2603e912a921: Status 404 returned error can't find the container with id 89de15d4e97898f119e105f0cda84642e2b0ab2314004a6eae8f2603e912a921
Oct 30 00:58:50.465119 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:50.465091    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8"]
Oct 30 00:58:50.493749 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:50.492621    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8" event={"ID":"7584e7cf-bf67-4082-adb9-415425b02f9d","Type":"ContainerStarted","Data":"89de15d4e97898f119e105f0cda84642e2b0ab2314004a6eae8f2603e912a921"}
Oct 30 00:58:51.504670 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:51.504637    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8" event={"ID":"7584e7cf-bf67-4082-adb9-415425b02f9d","Type":"ContainerStarted","Data":"38eef8fe5fc431b0e422dc7441cfb66179478c333750132ed2c0970eeb0da690"}
Oct 30 00:58:51.555655 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:51.555488    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8" podStartSLOduration=2.555426917 podStartE2EDuration="2.555426917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:51.55022051 +0000 UTC m=+1425.407128267" watchObservedRunningTime="2026-10-30 00:58:51.555426917 +0000 UTC m=+1425.412334672"
Oct 30 00:58:52.030916 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.030874    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s"]
Oct 30 00:58:52.042687 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.042650    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c07489-s4x7s"]
Oct 30 00:58:52.141222 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.141144    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf"]
Oct 30 00:58:52.145043 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.145025    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf"
Oct 30 00:58:52.164946 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.164920    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf"]
Oct 30 00:58:52.265310 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.265279    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40630740-9e4a-49cd-8aac-f4fcf9e4b378" path="/var/lib/kubelet/pods/40630740-9e4a-49cd-8aac-f4fcf9e4b378/volumes"
Oct 30 00:58:52.301246 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.301214    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d90b85-8170-47ab-bcdf-7d897eba3e7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf\" (UID: \"46d90b85-8170-47ab-bcdf-7d897eba3e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf"
Oct 30 00:58:52.301389 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.301256    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6ghc\" (UniqueName: \"kubernetes.io/projected/46d90b85-8170-47ab-bcdf-7d897eba3e7f-kube-api-access-p6ghc\") pod \"maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf\" (UID: \"46d90b85-8170-47ab-bcdf-7d897eba3e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf"
Oct 30 00:58:52.301449 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.301383    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d90b85-8170-47ab-bcdf-7d897eba3e7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf\" (UID: \"46d90b85-8170-47ab-bcdf-7d897eba3e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf"
Oct 30 00:58:52.403768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.403628    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d90b85-8170-47ab-bcdf-7d897eba3e7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf\" (UID: \"46d90b85-8170-47ab-bcdf-7d897eba3e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf"
Oct 30 00:58:52.403768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.403672    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6ghc\" (UniqueName: \"kubernetes.io/projected/46d90b85-8170-47ab-bcdf-7d897eba3e7f-kube-api-access-p6ghc\") pod \"maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf\" (UID: \"46d90b85-8170-47ab-bcdf-7d897eba3e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf"
Oct 30 00:58:52.403768 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.403727    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d90b85-8170-47ab-bcdf-7d897eba3e7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf\" (UID: \"46d90b85-8170-47ab-bcdf-7d897eba3e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf"
Oct 30 00:58:52.404454 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.404428    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d90b85-8170-47ab-bcdf-7d897eba3e7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf\" (UID: \"46d90b85-8170-47ab-bcdf-7d897eba3e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf"
Oct 30 00:58:52.406184 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.406164    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d90b85-8170-47ab-bcdf-7d897eba3e7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf\" (UID: \"46d90b85-8170-47ab-bcdf-7d897eba3e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf"
Oct 30 00:58:52.429005 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.428966    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6ghc\" (UniqueName: \"kubernetes.io/projected/46d90b85-8170-47ab-bcdf-7d897eba3e7f-kube-api-access-p6ghc\") pod \"maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf\" (UID: \"46d90b85-8170-47ab-bcdf-7d897eba3e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf"
Oct 30 00:58:52.459088 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.459050    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf"
Oct 30 00:58:52.518802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.518680    6099 generic.go:356] "Generic (PLEG): container finished" podID="7584e7cf-bf67-4082-adb9-415425b02f9d" containerID="38eef8fe5fc431b0e422dc7441cfb66179478c333750132ed2c0970eeb0da690" exitCode=0
Oct 30 00:58:52.518802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.518745    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8" event={"ID":"7584e7cf-bf67-4082-adb9-415425b02f9d","Type":"ContainerDied","Data":"38eef8fe5fc431b0e422dc7441cfb66179478c333750132ed2c0970eeb0da690"}
Oct 30 00:58:52.766894 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:58:52.766860    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d90b85_8170_47ab_bcdf_7d897eba3e7f.slice/crio-c8ca5a5ff4a006070e02cf708807d52fe68ca956f5a44441f605b6201d85f33e WatchSource:0}: Error finding container c8ca5a5ff4a006070e02cf708807d52fe68ca956f5a44441f605b6201d85f33e: Status 404 returned error can't find the container with id c8ca5a5ff4a006070e02cf708807d52fe68ca956f5a44441f605b6201d85f33e
Oct 30 00:58:52.768229 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:52.768193    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf"]
Oct 30 00:58:53.538180 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:53.537822    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf" event={"ID":"46d90b85-8170-47ab-bcdf-7d897eba3e7f","Type":"ContainerStarted","Data":"f5c1d7bffa355474fb32e673091be70baf903e4d960c3bbcd5e7435d4a72b430"}
Oct 30 00:58:53.538180 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:53.537914    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf" event={"ID":"46d90b85-8170-47ab-bcdf-7d897eba3e7f","Type":"ContainerStarted","Data":"c8ca5a5ff4a006070e02cf708807d52fe68ca956f5a44441f605b6201d85f33e"}
Oct 30 00:58:53.585855 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:53.585737    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf" podStartSLOduration=1.585721413 podStartE2EDuration="1.585721413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:53.57985485 +0000 UTC m=+1427.436762627" watchObservedRunningTime="2026-10-30 00:58:53.585721413 +0000 UTC m=+1427.442629164"
Oct 30 00:58:53.873154 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:53.873126    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8"
Oct 30 00:58:54.036353 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.036315    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7584e7cf-bf67-4082-adb9-415425b02f9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7584e7cf-bf67-4082-adb9-415425b02f9d-maas-api-xks-ca\") pod \"7584e7cf-bf67-4082-adb9-415425b02f9d\" (UID: \"7584e7cf-bf67-4082-adb9-415425b02f9d\") "
Oct 30 00:58:54.036353 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.036352    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7584e7cf-bf67-4082-adb9-415425b02f9d-kube-api-access-b9qhh\" (UniqueName: \"kubernetes.io/projected/7584e7cf-bf67-4082-adb9-415425b02f9d-kube-api-access-b9qhh\") pod \"7584e7cf-bf67-4082-adb9-415425b02f9d\" (UID: \"7584e7cf-bf67-4082-adb9-415425b02f9d\") "
Oct 30 00:58:54.036813 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.036792    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7584e7cf-bf67-4082-adb9-415425b02f9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7584e7cf-bf67-4082-adb9-415425b02f9d-maas-api-openshift-ca\") pod \"7584e7cf-bf67-4082-adb9-415425b02f9d\" (UID: \"7584e7cf-bf67-4082-adb9-415425b02f9d\") "
Oct 30 00:58:54.037216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.037189    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7584e7cf-bf67-4082-adb9-415425b02f9d-maas-api-openshift-ca" pod "7584e7cf-bf67-4082-adb9-415425b02f9d" (UID: "7584e7cf-bf67-4082-adb9-415425b02f9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:54.037646 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.037621    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7584e7cf-bf67-4082-adb9-415425b02f9d-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:54.038537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.038516    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7584e7cf-bf67-4082-adb9-415425b02f9d-kube-api-access-b9qhh" pod "7584e7cf-bf67-4082-adb9-415425b02f9d" (UID: "7584e7cf-bf67-4082-adb9-415425b02f9d"). InnerVolumeSpecName "kube-api-access-b9qhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:54.038671 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.038652    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7584e7cf-bf67-4082-adb9-415425b02f9d-maas-api-xks-ca" pod "7584e7cf-bf67-4082-adb9-415425b02f9d" (UID: "7584e7cf-bf67-4082-adb9-415425b02f9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:54.139149 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.139113    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7584e7cf-bf67-4082-adb9-415425b02f9d-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:54.139149 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.139145    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9qhh\" (UniqueName: \"kubernetes.io/projected/7584e7cf-bf67-4082-adb9-415425b02f9d-kube-api-access-b9qhh\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:54.547982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.547953    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8"
Oct 30 00:58:54.547982 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.547968    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8" event={"ID":"7584e7cf-bf67-4082-adb9-415425b02f9d","Type":"ContainerDied","Data":"89de15d4e97898f119e105f0cda84642e2b0ab2314004a6eae8f2603e912a921"}
Oct 30 00:58:54.548420 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.548017    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89de15d4e97898f119e105f0cda84642e2b0ab2314004a6eae8f2603e912a921"
Oct 30 00:58:54.549859 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.549825    6099 generic.go:356] "Generic (PLEG): container finished" podID="46d90b85-8170-47ab-bcdf-7d897eba3e7f" containerID="f5c1d7bffa355474fb32e673091be70baf903e4d960c3bbcd5e7435d4a72b430" exitCode=0
Oct 30 00:58:54.549989 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.549888    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf" event={"ID":"46d90b85-8170-47ab-bcdf-7d897eba3e7f","Type":"ContainerDied","Data":"f5c1d7bffa355474fb32e673091be70baf903e4d960c3bbcd5e7435d4a72b430"}
Oct 30 00:58:54.614508 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.614476    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4"]
Oct 30 00:58:54.615946 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.615903    6099 cpu_manager.go:414] "containerMap: removing container" podUID="7584e7cf-bf67-4082-adb9-415425b02f9d" containerName="revoke-keys"
Oct 30 00:58:54.615946 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.615944    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7584e7cf-bf67-4082-adb9-415425b02f9d" containerName="revoke-keys"
Oct 30 00:58:54.616140 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.616128    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7584e7cf-bf67-4082-adb9-415425b02f9d" containerName="revoke-keys"
Oct 30 00:58:54.620556 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.620535    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4"
Oct 30 00:58:54.650903 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.648157    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44p8j\" (UniqueName: \"kubernetes.io/projected/d5be716f-50a7-4723-acf5-9b987bf7fab8-kube-api-access-44p8j\") pod \"maas-api-revoke-keys-e2e-api-a-353403-z2ck4\" (UID: \"d5be716f-50a7-4723-acf5-9b987bf7fab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4"
Oct 30 00:58:54.650903 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.648230    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5be716f-50a7-4723-acf5-9b987bf7fab8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-353403-z2ck4\" (UID: \"d5be716f-50a7-4723-acf5-9b987bf7fab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4"
Oct 30 00:58:54.650903 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.648399    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5be716f-50a7-4723-acf5-9b987bf7fab8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-353403-z2ck4\" (UID: \"d5be716f-50a7-4723-acf5-9b987bf7fab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4"
Oct 30 00:58:54.654518 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.654492    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4"]
Oct 30 00:58:54.751436 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.751400    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5be716f-50a7-4723-acf5-9b987bf7fab8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-353403-z2ck4\" (UID: \"d5be716f-50a7-4723-acf5-9b987bf7fab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4"
Oct 30 00:58:54.751680 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.751661    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5be716f-50a7-4723-acf5-9b987bf7fab8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-353403-z2ck4\" (UID: \"d5be716f-50a7-4723-acf5-9b987bf7fab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4"
Oct 30 00:58:54.751987 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.751961    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44p8j\" (UniqueName: \"kubernetes.io/projected/d5be716f-50a7-4723-acf5-9b987bf7fab8-kube-api-access-44p8j\") pod \"maas-api-revoke-keys-e2e-api-a-353403-z2ck4\" (UID: \"d5be716f-50a7-4723-acf5-9b987bf7fab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4"
Oct 30 00:58:54.752344 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.752309    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5be716f-50a7-4723-acf5-9b987bf7fab8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-353403-z2ck4\" (UID: \"d5be716f-50a7-4723-acf5-9b987bf7fab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4"
Oct 30 00:58:54.753954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.753932    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5be716f-50a7-4723-acf5-9b987bf7fab8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-353403-z2ck4\" (UID: \"d5be716f-50a7-4723-acf5-9b987bf7fab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4"
Oct 30 00:58:54.774353 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.774331    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44p8j\" (UniqueName: \"kubernetes.io/projected/d5be716f-50a7-4723-acf5-9b987bf7fab8-kube-api-access-44p8j\") pod \"maas-api-revoke-keys-e2e-api-a-353403-z2ck4\" (UID: \"d5be716f-50a7-4723-acf5-9b987bf7fab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4"
Oct 30 00:58:54.935844 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:54.935762    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4"
Oct 30 00:58:55.243514 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:58:55.243480    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5be716f_50a7_4723_acf5_9b987bf7fab8.slice/crio-2ebfa152127acd69b05407a8f0752c0c32d2dafcb2ea5e02bd63b49a8ab0223f WatchSource:0}: Error finding container 2ebfa152127acd69b05407a8f0752c0c32d2dafcb2ea5e02bd63b49a8ab0223f: Status 404 returned error can't find the container with id 2ebfa152127acd69b05407a8f0752c0c32d2dafcb2ea5e02bd63b49a8ab0223f
Oct 30 00:58:55.259748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:55.245945    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4"]
Oct 30 00:58:55.562147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:55.562116    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4" event={"ID":"d5be716f-50a7-4723-acf5-9b987bf7fab8","Type":"ContainerStarted","Data":"2ebfa152127acd69b05407a8f0752c0c32d2dafcb2ea5e02bd63b49a8ab0223f"}
Oct 30 00:58:55.764353 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:55.764291    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf"
Oct 30 00:58:55.887320 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:55.887289    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46d90b85-8170-47ab-bcdf-7d897eba3e7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d90b85-8170-47ab-bcdf-7d897eba3e7f-maas-api-xks-ca\") pod \"46d90b85-8170-47ab-bcdf-7d897eba3e7f\" (UID: \"46d90b85-8170-47ab-bcdf-7d897eba3e7f\") "
Oct 30 00:58:55.887641 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:55.887616    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d90b85-8170-47ab-bcdf-7d897eba3e7f-kube-api-access-p6ghc\" (UniqueName: \"kubernetes.io/projected/46d90b85-8170-47ab-bcdf-7d897eba3e7f-kube-api-access-p6ghc\") pod \"46d90b85-8170-47ab-bcdf-7d897eba3e7f\" (UID: \"46d90b85-8170-47ab-bcdf-7d897eba3e7f\") "
Oct 30 00:58:55.887753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:55.887688    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d90b85-8170-47ab-bcdf-7d897eba3e7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d90b85-8170-47ab-bcdf-7d897eba3e7f-maas-api-openshift-ca\") pod \"46d90b85-8170-47ab-bcdf-7d897eba3e7f\" (UID: \"46d90b85-8170-47ab-bcdf-7d897eba3e7f\") "
Oct 30 00:58:55.888041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:55.888017    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d90b85-8170-47ab-bcdf-7d897eba3e7f-maas-api-openshift-ca" pod "46d90b85-8170-47ab-bcdf-7d897eba3e7f" (UID: "46d90b85-8170-47ab-bcdf-7d897eba3e7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:55.888588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:55.888571    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d90b85-8170-47ab-bcdf-7d897eba3e7f-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:55.889494 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:55.889469    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d90b85-8170-47ab-bcdf-7d897eba3e7f-maas-api-xks-ca" pod "46d90b85-8170-47ab-bcdf-7d897eba3e7f" (UID: "46d90b85-8170-47ab-bcdf-7d897eba3e7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:55.889815 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:55.889796    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d90b85-8170-47ab-bcdf-7d897eba3e7f-kube-api-access-p6ghc" pod "46d90b85-8170-47ab-bcdf-7d897eba3e7f" (UID: "46d90b85-8170-47ab-bcdf-7d897eba3e7f"). InnerVolumeSpecName "kube-api-access-p6ghc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:55.990631 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:55.990599    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d90b85-8170-47ab-bcdf-7d897eba3e7f-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:55.990631 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:55.990629    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6ghc\" (UniqueName: \"kubernetes.io/projected/46d90b85-8170-47ab-bcdf-7d897eba3e7f-kube-api-access-p6ghc\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:56.572662 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:56.572629    6099 generic.go:356] "Generic (PLEG): container finished" podID="d5be716f-50a7-4723-acf5-9b987bf7fab8" containerID="f34d797730927935a2ca15215509d10cc28faeb785d849e25284c35818ca9082" exitCode=0
Oct 30 00:58:56.573221 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:56.572672    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4" event={"ID":"d5be716f-50a7-4723-acf5-9b987bf7fab8","Type":"ContainerDied","Data":"f34d797730927935a2ca15215509d10cc28faeb785d849e25284c35818ca9082"}
Oct 30 00:58:56.574217 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:56.574200    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf"
Oct 30 00:58:56.574333 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:56.574244    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf" event={"ID":"46d90b85-8170-47ab-bcdf-7d897eba3e7f","Type":"ContainerDied","Data":"c8ca5a5ff4a006070e02cf708807d52fe68ca956f5a44441f605b6201d85f33e"}
Oct 30 00:58:56.574333 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:56.574295    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8ca5a5ff4a006070e02cf708807d52fe68ca956f5a44441f605b6201d85f33e"
Oct 30 00:58:57.766251 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.766224    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4"
Oct 30 00:58:57.822997 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.822963    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5be716f-50a7-4723-acf5-9b987bf7fab8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5be716f-50a7-4723-acf5-9b987bf7fab8-maas-api-openshift-ca\") pod \"d5be716f-50a7-4723-acf5-9b987bf7fab8\" (UID: \"d5be716f-50a7-4723-acf5-9b987bf7fab8\") "
Oct 30 00:58:57.823185 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.823167    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5be716f-50a7-4723-acf5-9b987bf7fab8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5be716f-50a7-4723-acf5-9b987bf7fab8-maas-api-xks-ca\") pod \"d5be716f-50a7-4723-acf5-9b987bf7fab8\" (UID: \"d5be716f-50a7-4723-acf5-9b987bf7fab8\") "
Oct 30 00:58:57.823245 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.823212    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5be716f-50a7-4723-acf5-9b987bf7fab8-kube-api-access-44p8j\" (UniqueName: \"kubernetes.io/projected/d5be716f-50a7-4723-acf5-9b987bf7fab8-kube-api-access-44p8j\") pod \"d5be716f-50a7-4723-acf5-9b987bf7fab8\" (UID: \"d5be716f-50a7-4723-acf5-9b987bf7fab8\") "
Oct 30 00:58:57.823341 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.823314    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5be716f-50a7-4723-acf5-9b987bf7fab8-maas-api-openshift-ca" pod "d5be716f-50a7-4723-acf5-9b987bf7fab8" (UID: "d5be716f-50a7-4723-acf5-9b987bf7fab8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:57.824173 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.824146    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5be716f-50a7-4723-acf5-9b987bf7fab8-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:57.825361 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.825332    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5be716f-50a7-4723-acf5-9b987bf7fab8-maas-api-xks-ca" pod "d5be716f-50a7-4723-acf5-9b987bf7fab8" (UID: "d5be716f-50a7-4723-acf5-9b987bf7fab8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:57.825547 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.825531    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5be716f-50a7-4723-acf5-9b987bf7fab8-kube-api-access-44p8j" pod "d5be716f-50a7-4723-acf5-9b987bf7fab8" (UID: "d5be716f-50a7-4723-acf5-9b987bf7fab8"). InnerVolumeSpecName "kube-api-access-44p8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:57.926477 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.926444    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5be716f-50a7-4723-acf5-9b987bf7fab8-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:57.926477 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.926475    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44p8j\" (UniqueName: \"kubernetes.io/projected/d5be716f-50a7-4723-acf5-9b987bf7fab8-kube-api-access-44p8j\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:58:57.944416 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.944382    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt"]
Oct 30 00:58:57.945326 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.945303    6099 cpu_manager.go:414] "containerMap: removing container" podUID="46d90b85-8170-47ab-bcdf-7d897eba3e7f" containerName="revoke-keys"
Oct 30 00:58:57.945326 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.945322    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d90b85-8170-47ab-bcdf-7d897eba3e7f" containerName="revoke-keys"
Oct 30 00:58:57.945524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.945336    6099 cpu_manager.go:414] "containerMap: removing container" podUID="d5be716f-50a7-4723-acf5-9b987bf7fab8" containerName="revoke-keys"
Oct 30 00:58:57.945524 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.945342    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5be716f-50a7-4723-acf5-9b987bf7fab8" containerName="revoke-keys"
Oct 30 00:58:57.945636 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.945519    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d90b85-8170-47ab-bcdf-7d897eba3e7f" containerName="revoke-keys"
Oct 30 00:58:57.945636 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.945558    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5be716f-50a7-4723-acf5-9b987bf7fab8" containerName="revoke-keys"
Oct 30 00:58:57.949613 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.949591    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt"
Oct 30 00:58:57.968527 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:57.968500    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt"]
Oct 30 00:58:58.027676 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:58.027647    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b852a375-9dcb-4094-88b0-dca00f669314-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt\" (UID: \"b852a375-9dcb-4094-88b0-dca00f669314\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt"
Oct 30 00:58:58.027880 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:58.027865    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b852a375-9dcb-4094-88b0-dca00f669314-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt\" (UID: \"b852a375-9dcb-4094-88b0-dca00f669314\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt"
Oct 30 00:58:58.028000 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:58.027980    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz269\" (UniqueName: \"kubernetes.io/projected/b852a375-9dcb-4094-88b0-dca00f669314-kube-api-access-kz269\") pod \"maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt\" (UID: \"b852a375-9dcb-4094-88b0-dca00f669314\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt"
Oct 30 00:58:58.129291 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:58.129202    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz269\" (UniqueName: \"kubernetes.io/projected/b852a375-9dcb-4094-88b0-dca00f669314-kube-api-access-kz269\") pod \"maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt\" (UID: \"b852a375-9dcb-4094-88b0-dca00f669314\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt"
Oct 30 00:58:58.129453 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:58.129368    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b852a375-9dcb-4094-88b0-dca00f669314-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt\" (UID: \"b852a375-9dcb-4094-88b0-dca00f669314\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt"
Oct 30 00:58:58.129527 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:58.129511    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b852a375-9dcb-4094-88b0-dca00f669314-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt\" (UID: \"b852a375-9dcb-4094-88b0-dca00f669314\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt"
Oct 30 00:58:58.130092 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:58.130065    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b852a375-9dcb-4094-88b0-dca00f669314-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt\" (UID: \"b852a375-9dcb-4094-88b0-dca00f669314\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt"
Oct 30 00:58:58.131865 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:58.131844    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b852a375-9dcb-4094-88b0-dca00f669314-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt\" (UID: \"b852a375-9dcb-4094-88b0-dca00f669314\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt"
Oct 30 00:58:58.152129 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:58.152105    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz269\" (UniqueName: \"kubernetes.io/projected/b852a375-9dcb-4094-88b0-dca00f669314-kube-api-access-kz269\") pod \"maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt\" (UID: \"b852a375-9dcb-4094-88b0-dca00f669314\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt"
Oct 30 00:58:58.271680 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:58.271647    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt"
Oct 30 00:58:58.582665 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:58:58.582635    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb852a375_9dcb_4094_88b0_dca00f669314.slice/crio-2031ce79ae9169979d66d68e61033eaff7d117c1f9f07699ffd5ba8c3821b113 WatchSource:0}: Error finding container 2031ce79ae9169979d66d68e61033eaff7d117c1f9f07699ffd5ba8c3821b113: Status 404 returned error can't find the container with id 2031ce79ae9169979d66d68e61033eaff7d117c1f9f07699ffd5ba8c3821b113
Oct 30 00:58:58.584529 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:58.584319    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt"]
Oct 30 00:58:58.595638 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:58.595614    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt" event={"ID":"b852a375-9dcb-4094-88b0-dca00f669314","Type":"ContainerStarted","Data":"2031ce79ae9169979d66d68e61033eaff7d117c1f9f07699ffd5ba8c3821b113"}
Oct 30 00:58:58.597186 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:58.597166    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4"
Oct 30 00:58:58.597281 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:58.597192    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4" event={"ID":"d5be716f-50a7-4723-acf5-9b987bf7fab8","Type":"ContainerDied","Data":"2ebfa152127acd69b05407a8f0752c0c32d2dafcb2ea5e02bd63b49a8ab0223f"}
Oct 30 00:58:58.597281 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:58.597242    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ebfa152127acd69b05407a8f0752c0c32d2dafcb2ea5e02bd63b49a8ab0223f"
Oct 30 00:58:59.635869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:59.622152    6099 generic.go:356] "Generic (PLEG): container finished" podID="b852a375-9dcb-4094-88b0-dca00f669314" containerID="1d16df1e835e2d935cffd7f1de1d5dabcaa306c09d12ce2975002d795dd9c331" exitCode=0
Oct 30 00:58:59.635869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:58:59.623245    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt" event={"ID":"b852a375-9dcb-4094-88b0-dca00f669314","Type":"ContainerDied","Data":"1d16df1e835e2d935cffd7f1de1d5dabcaa306c09d12ce2975002d795dd9c331"}
Oct 30 00:59:00.041087 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:00.040480    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:59:00.834783 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:00.834759    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt"
Oct 30 00:59:00.850272 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:00.850244    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b852a375-9dcb-4094-88b0-dca00f669314-kube-api-access-kz269\" (UniqueName: \"kubernetes.io/projected/b852a375-9dcb-4094-88b0-dca00f669314-kube-api-access-kz269\") pod \"b852a375-9dcb-4094-88b0-dca00f669314\" (UID: \"b852a375-9dcb-4094-88b0-dca00f669314\") "
Oct 30 00:59:00.850511 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:00.850492    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b852a375-9dcb-4094-88b0-dca00f669314-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b852a375-9dcb-4094-88b0-dca00f669314-maas-api-xks-ca\") pod \"b852a375-9dcb-4094-88b0-dca00f669314\" (UID: \"b852a375-9dcb-4094-88b0-dca00f669314\") "
Oct 30 00:59:00.850744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:00.850721    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b852a375-9dcb-4094-88b0-dca00f669314-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b852a375-9dcb-4094-88b0-dca00f669314-maas-api-openshift-ca\") pod \"b852a375-9dcb-4094-88b0-dca00f669314\" (UID: \"b852a375-9dcb-4094-88b0-dca00f669314\") "
Oct 30 00:59:00.851166 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:00.851139    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b852a375-9dcb-4094-88b0-dca00f669314-maas-api-openshift-ca" pod "b852a375-9dcb-4094-88b0-dca00f669314" (UID: "b852a375-9dcb-4094-88b0-dca00f669314"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:00.851606 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:00.851551    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b852a375-9dcb-4094-88b0-dca00f669314-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:00.852538 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:00.852514    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b852a375-9dcb-4094-88b0-dca00f669314-kube-api-access-kz269" pod "b852a375-9dcb-4094-88b0-dca00f669314" (UID: "b852a375-9dcb-4094-88b0-dca00f669314"). InnerVolumeSpecName "kube-api-access-kz269". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:00.852748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:00.852693    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b852a375-9dcb-4094-88b0-dca00f669314-maas-api-xks-ca" pod "b852a375-9dcb-4094-88b0-dca00f669314" (UID: "b852a375-9dcb-4094-88b0-dca00f669314"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:59:00.953761 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:00.953661    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz269\" (UniqueName: \"kubernetes.io/projected/b852a375-9dcb-4094-88b0-dca00f669314-kube-api-access-kz269\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:00.953761 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:00.953690    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b852a375-9dcb-4094-88b0-dca00f669314-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:01.643085 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:01.643045    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt"
Oct 30 00:59:01.643085 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:01.643067    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt" event={"ID":"b852a375-9dcb-4094-88b0-dca00f669314","Type":"ContainerDied","Data":"2031ce79ae9169979d66d68e61033eaff7d117c1f9f07699ffd5ba8c3821b113"}
Oct 30 00:59:01.643307 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:01.643122    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2031ce79ae9169979d66d68e61033eaff7d117c1f9f07699ffd5ba8c3821b113"
Oct 30 00:59:02.645684 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:02.645648    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv"]
Oct 30 00:59:02.680258 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:02.680223    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq"]
Oct 30 00:59:05.035869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:05.035831    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt"]
Oct 30 00:59:05.085579 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:05.085544    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx"]
Oct 30 00:59:05.096274 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:05.096237    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"]
Oct 30 00:59:05.096485 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:05.096457    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l" podUID="2bd5a458-625c-4eec-b0df-bbecc83ebed4" containerName="istio-proxy" containerID="cri-o://564bb0e684e51330212851e56a666ce6eeee2fb176b3384586b9bf7015a41ff8" gracePeriod=30
Oct 30 00:59:07.646154 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:07.646100    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv" podUID="42140e21-60da-460b-8d55-d7b9f5e53a2d" containerName="payload-processing" containerID="cri-o://a1f4f27955508b704b853f02dcc79108e1569d3441e9fa8a9fe917e920b73b73" gracePeriod=25
Oct 30 00:59:07.680921 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:07.680874    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq" podUID="5ec09452-0400-491f-b1df-65d1e4160023" containerName="payload-pre-processing" containerID="cri-o://6797937017abb82841618acd3ea9f4d4c70e29f26ea77e8db87767c874402adc" gracePeriod=25
Oct 30 00:59:07.767676 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:07.767639    6099 generic.go:356] "Generic (PLEG): container finished" podID="42140e21-60da-460b-8d55-d7b9f5e53a2d" containerID="a1f4f27955508b704b853f02dcc79108e1569d3441e9fa8a9fe917e920b73b73" exitCode=0
Oct 30 00:59:07.767852 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:07.767722    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv" event={"ID":"42140e21-60da-460b-8d55-d7b9f5e53a2d","Type":"ContainerDied","Data":"a1f4f27955508b704b853f02dcc79108e1569d3441e9fa8a9fe917e920b73b73"}
Oct 30 00:59:08.008698 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.008671    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv"
Oct 30 00:59:08.037927 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.037899    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq"
Oct 30 00:59:08.175382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.172400    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42140e21-60da-460b-8d55-d7b9f5e53a2d-kube-api-access-4p7f6\" (UniqueName: \"kubernetes.io/projected/42140e21-60da-460b-8d55-d7b9f5e53a2d-kube-api-access-4p7f6\") pod \"42140e21-60da-460b-8d55-d7b9f5e53a2d\" (UID: \"42140e21-60da-460b-8d55-d7b9f5e53a2d\") "
Oct 30 00:59:08.175382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.172461    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42140e21-60da-460b-8d55-d7b9f5e53a2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42140e21-60da-460b-8d55-d7b9f5e53a2d-plugins-config-volume\") pod \"42140e21-60da-460b-8d55-d7b9f5e53a2d\" (UID: \"42140e21-60da-460b-8d55-d7b9f5e53a2d\") "
Oct 30 00:59:08.175382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.172803    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ec09452-0400-491f-b1df-65d1e4160023-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec09452-0400-491f-b1df-65d1e4160023-plugins-config-volume\") pod \"5ec09452-0400-491f-b1df-65d1e4160023\" (UID: \"5ec09452-0400-491f-b1df-65d1e4160023\") "
Oct 30 00:59:08.175382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.172994    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec09452-0400-491f-b1df-65d1e4160023-kube-api-access-44h4t\" (UniqueName: \"kubernetes.io/projected/5ec09452-0400-491f-b1df-65d1e4160023-kube-api-access-44h4t\") pod \"5ec09452-0400-491f-b1df-65d1e4160023\" (UID: \"5ec09452-0400-491f-b1df-65d1e4160023\") "
Oct 30 00:59:08.175382 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.174766    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42140e21-60da-460b-8d55-d7b9f5e53a2d-plugins-config-volume" pod "42140e21-60da-460b-8d55-d7b9f5e53a2d" (UID: "42140e21-60da-460b-8d55-d7b9f5e53a2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:08.175839 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.175525    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec09452-0400-491f-b1df-65d1e4160023-plugins-config-volume" pod "5ec09452-0400-491f-b1df-65d1e4160023" (UID: "5ec09452-0400-491f-b1df-65d1e4160023"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:08.176941 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.176648    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec09452-0400-491f-b1df-65d1e4160023-kube-api-access-44h4t" pod "5ec09452-0400-491f-b1df-65d1e4160023" (UID: "5ec09452-0400-491f-b1df-65d1e4160023"). InnerVolumeSpecName "kube-api-access-44h4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:08.178062 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.178041    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42140e21-60da-460b-8d55-d7b9f5e53a2d-kube-api-access-4p7f6" pod "42140e21-60da-460b-8d55-d7b9f5e53a2d" (UID: "42140e21-60da-460b-8d55-d7b9f5e53a2d"). InnerVolumeSpecName "kube-api-access-4p7f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:08.275313 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.275281    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec09452-0400-491f-b1df-65d1e4160023-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:08.275313 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.275312    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44h4t\" (UniqueName: \"kubernetes.io/projected/5ec09452-0400-491f-b1df-65d1e4160023-kube-api-access-44h4t\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:08.275471 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.275321    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p7f6\" (UniqueName: \"kubernetes.io/projected/42140e21-60da-460b-8d55-d7b9f5e53a2d-kube-api-access-4p7f6\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:08.275471 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.275330    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42140e21-60da-460b-8d55-d7b9f5e53a2d-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:08.778932 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.778894    6099 generic.go:356] "Generic (PLEG): container finished" podID="5ec09452-0400-491f-b1df-65d1e4160023" containerID="6797937017abb82841618acd3ea9f4d4c70e29f26ea77e8db87767c874402adc" exitCode=0
Oct 30 00:59:08.779339 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.778952    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq"
Oct 30 00:59:08.779339 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.778945    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq" event={"ID":"5ec09452-0400-491f-b1df-65d1e4160023","Type":"ContainerDied","Data":"6797937017abb82841618acd3ea9f4d4c70e29f26ea77e8db87767c874402adc"}
Oct 30 00:59:08.779339 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.779109    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq" event={"ID":"5ec09452-0400-491f-b1df-65d1e4160023","Type":"ContainerDied","Data":"b5fd101607e76326647849ab84d4097eaba9b27927259a99c73304e399c3c010"}
Oct 30 00:59:08.779339 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.779138    6099 scope.go:122] "RemoveContainer" containerID="6797937017abb82841618acd3ea9f4d4c70e29f26ea77e8db87767c874402adc"
Oct 30 00:59:08.780650 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.780625    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv"
Oct 30 00:59:08.780822 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.780796    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv" event={"ID":"42140e21-60da-460b-8d55-d7b9f5e53a2d","Type":"ContainerDied","Data":"ed62f9f3a6c9582db7a8e5bad789bb3db123de283170bc283005605f864eae83"}
Oct 30 00:59:08.792594 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.792576    6099 scope.go:122] "RemoveContainer" containerID="6797937017abb82841618acd3ea9f4d4c70e29f26ea77e8db87767c874402adc"
Oct 30 00:59:08.792906 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:59:08.792885    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6797937017abb82841618acd3ea9f4d4c70e29f26ea77e8db87767c874402adc\": container with ID starting with 6797937017abb82841618acd3ea9f4d4c70e29f26ea77e8db87767c874402adc not found: ID does not exist" containerID="6797937017abb82841618acd3ea9f4d4c70e29f26ea77e8db87767c874402adc"
Oct 30 00:59:08.792961 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.792912    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6797937017abb82841618acd3ea9f4d4c70e29f26ea77e8db87767c874402adc"} err="failed to get container status \"6797937017abb82841618acd3ea9f4d4c70e29f26ea77e8db87767c874402adc\": rpc error: code = NotFound desc = could not find container \"6797937017abb82841618acd3ea9f4d4c70e29f26ea77e8db87767c874402adc\": container with ID starting with 6797937017abb82841618acd3ea9f4d4c70e29f26ea77e8db87767c874402adc not found: ID does not exist"
Oct 30 00:59:08.792961 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.792928    6099 scope.go:122] "RemoveContainer" containerID="a1f4f27955508b704b853f02dcc79108e1569d3441e9fa8a9fe917e920b73b73"
Oct 30 00:59:08.838367 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.838337    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq"]
Oct 30 00:59:08.850412 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.850385    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73b32e-7dd75d8667-jxhrq"]
Oct 30 00:59:08.873319 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.873284    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv"]
Oct 30 00:59:08.877536 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:08.877512    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73b32e-6655649548-fn9bv"]
Oct 30 00:59:10.036493 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.036450    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt" podUID="984aa715-0b1d-450d-a28c-cc85e4beb848" containerName="payload-processing" containerID="cri-o://b5ea418f68adf5148147398c6ff891c14c4fbd0c1691cc658a9359bf42b43881" gracePeriod=25
Oct 30 00:59:10.086631 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.086600    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx" podUID="d6a88bf7-5dfd-4a6b-827a-4dbad7d58828" containerName="payload-pre-processing" containerID="cri-o://149e182606a19627d05ec99ce7af8a8987fd988d7cbec4916c21c92839b2d87f" gracePeriod=25
Oct 30 00:59:10.191594 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.191560    6099 scope.go:122] "RemoveContainer" containerID="7506e3000c8c947cd74aec5505a4fa4163a469e6d9a589307e69a52ff618b356"
Oct 30 00:59:10.232634 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.232590    6099 scope.go:122] "RemoveContainer" containerID="ffee9dde426f61747e891c4b89ec7fe751d9bfa22e98f183735fe38f2a081203"
Oct 30 00:59:10.251450 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.251362    6099 scope.go:122] "RemoveContainer" containerID="38c473367134ee20adf0969ac32ed654a1c2c0530025cc5545a696fcbcad1696"
Oct 30 00:59:10.279143 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.279031    6099 scope.go:122] "RemoveContainer" containerID="8a5da39bad16026be0bbd8703a25f5bbadde54305a817b1af57e8735a8e84834"
Oct 30 00:59:10.289369 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.289021    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42140e21-60da-460b-8d55-d7b9f5e53a2d" path="/var/lib/kubelet/pods/42140e21-60da-460b-8d55-d7b9f5e53a2d/volumes"
Oct 30 00:59:10.292788 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.292763    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec09452-0400-491f-b1df-65d1e4160023" path="/var/lib/kubelet/pods/5ec09452-0400-491f-b1df-65d1e4160023/volumes"
Oct 30 00:59:10.312405 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.312370    6099 scope.go:122] "RemoveContainer" containerID="44be71875a54fdc4e89300e5ca0b69e30cc31a22e7044c0ca046c6858bf80bf7"
Oct 30 00:59:10.341939 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.338551    6099 scope.go:122] "RemoveContainer" containerID="a601bf87e6ede97c4ec08e6c0b074506e21e7aaae97980e77e191bd31334a344"
Oct 30 00:59:10.473817 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.473791    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt"
Oct 30 00:59:10.556655 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.556604    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx"
Oct 30 00:59:10.560846 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.560826    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:59:10.603692 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.603664    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984aa715-0b1d-450d-a28c-cc85e4beb848-kube-api-access-dqz2c\" (UniqueName: \"kubernetes.io/projected/984aa715-0b1d-450d-a28c-cc85e4beb848-kube-api-access-dqz2c\") pod \"984aa715-0b1d-450d-a28c-cc85e4beb848\" (UID: \"984aa715-0b1d-450d-a28c-cc85e4beb848\") "
Oct 30 00:59:10.603905 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.603851    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-workload-certs\") pod \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") "
Oct 30 00:59:10.603987 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.603937    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-token\" (UniqueName: \"kubernetes.io/projected/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-token\") pod \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") "
Oct 30 00:59:10.603987 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.603967    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-workload-socket\") pod \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") "
Oct 30 00:59:10.604300 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.604088    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a88bf7-5dfd-4a6b-827a-4dbad7d58828-kube-api-access-jff2j\" (UniqueName: \"kubernetes.io/projected/d6a88bf7-5dfd-4a6b-827a-4dbad7d58828-kube-api-access-jff2j\") pod \"d6a88bf7-5dfd-4a6b-827a-4dbad7d58828\" (UID: \"d6a88bf7-5dfd-4a6b-827a-4dbad7d58828\") "
Oct 30 00:59:10.604300 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.604284    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-data\") pod \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") "
Oct 30 00:59:10.604463 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.604324    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-podinfo\") pod \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") "
Oct 30 00:59:10.604463 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.604379    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6a88bf7-5dfd-4a6b-827a-4dbad7d58828-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a88bf7-5dfd-4a6b-827a-4dbad7d58828-plugins-config-volume\") pod \"d6a88bf7-5dfd-4a6b-827a-4dbad7d58828\" (UID: \"d6a88bf7-5dfd-4a6b-827a-4dbad7d58828\") "
Oct 30 00:59:10.604571 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.604459    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-credential-socket\") pod \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") "
Oct 30 00:59:10.604571 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.604501    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/984aa715-0b1d-450d-a28c-cc85e4beb848-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984aa715-0b1d-450d-a28c-cc85e4beb848-plugins-config-volume\") pod \"984aa715-0b1d-450d-a28c-cc85e4beb848\" (UID: \"984aa715-0b1d-450d-a28c-cc85e4beb848\") "
Oct 30 00:59:10.604571 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.604562    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd5a458-625c-4eec-b0df-bbecc83ebed4-kube-api-access-9vf9g\" (UniqueName: \"kubernetes.io/projected/2bd5a458-625c-4eec-b0df-bbecc83ebed4-kube-api-access-9vf9g\") pod \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") "
Oct 30 00:59:10.604735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.604590    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istiod-ca-cert\") pod \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") "
Oct 30 00:59:10.604735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.604620    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-envoy\") pod \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\" (UID: \"2bd5a458-625c-4eec-b0df-bbecc83ebed4\") "
Oct 30 00:59:10.606046 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.604876    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-workload-certs" pod "2bd5a458-625c-4eec-b0df-bbecc83ebed4" (UID: "2bd5a458-625c-4eec-b0df-bbecc83ebed4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:10.606046 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.605732    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:10.606965 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.606612    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-workload-socket" pod "2bd5a458-625c-4eec-b0df-bbecc83ebed4" (UID: "2bd5a458-625c-4eec-b0df-bbecc83ebed4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:10.606965 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.606835    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984aa715-0b1d-450d-a28c-cc85e4beb848-plugins-config-volume" pod "984aa715-0b1d-450d-a28c-cc85e4beb848" (UID: "984aa715-0b1d-450d-a28c-cc85e4beb848"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:10.606965 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.606928    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a88bf7-5dfd-4a6b-827a-4dbad7d58828-plugins-config-volume" pod "d6a88bf7-5dfd-4a6b-827a-4dbad7d58828" (UID: "d6a88bf7-5dfd-4a6b-827a-4dbad7d58828"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:10.607251 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.607038    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-credential-socket" pod "2bd5a458-625c-4eec-b0df-bbecc83ebed4" (UID: "2bd5a458-625c-4eec-b0df-bbecc83ebed4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:10.608105 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.607487    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-data" pod "2bd5a458-625c-4eec-b0df-bbecc83ebed4" (UID: "2bd5a458-625c-4eec-b0df-bbecc83ebed4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:10.608105 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.607603    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-envoy" pod "2bd5a458-625c-4eec-b0df-bbecc83ebed4" (UID: "2bd5a458-625c-4eec-b0df-bbecc83ebed4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:10.608105 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.608060    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istiod-ca-cert" pod "2bd5a458-625c-4eec-b0df-bbecc83ebed4" (UID: "2bd5a458-625c-4eec-b0df-bbecc83ebed4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:10.609920 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.609898    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-podinfo" pod "2bd5a458-625c-4eec-b0df-bbecc83ebed4" (UID: "2bd5a458-625c-4eec-b0df-bbecc83ebed4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:10.610039 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.609961    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a88bf7-5dfd-4a6b-827a-4dbad7d58828-kube-api-access-jff2j" pod "d6a88bf7-5dfd-4a6b-827a-4dbad7d58828" (UID: "d6a88bf7-5dfd-4a6b-827a-4dbad7d58828"). InnerVolumeSpecName "kube-api-access-jff2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:10.610157 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.610138    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-token" pod "2bd5a458-625c-4eec-b0df-bbecc83ebed4" (UID: "2bd5a458-625c-4eec-b0df-bbecc83ebed4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:10.610675 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.610644    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd5a458-625c-4eec-b0df-bbecc83ebed4-kube-api-access-9vf9g" pod "2bd5a458-625c-4eec-b0df-bbecc83ebed4" (UID: "2bd5a458-625c-4eec-b0df-bbecc83ebed4"). InnerVolumeSpecName "kube-api-access-9vf9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:10.611209 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.611185    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984aa715-0b1d-450d-a28c-cc85e4beb848-kube-api-access-dqz2c" pod "984aa715-0b1d-450d-a28c-cc85e4beb848" (UID: "984aa715-0b1d-450d-a28c-cc85e4beb848"). InnerVolumeSpecName "kube-api-access-dqz2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:10.708391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.708354    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:10.708391 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.708392    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:10.708616 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.708407    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jff2j\" (UniqueName: \"kubernetes.io/projected/d6a88bf7-5dfd-4a6b-827a-4dbad7d58828-kube-api-access-jff2j\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:10.708616 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.708425    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:10.708616 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.708439    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:10.708616 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.708452    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a88bf7-5dfd-4a6b-827a-4dbad7d58828-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:10.708616 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.708465    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:10.708616 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.708477    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984aa715-0b1d-450d-a28c-cc85e4beb848-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:10.708616 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.708491    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vf9g\" (UniqueName: \"kubernetes.io/projected/2bd5a458-625c-4eec-b0df-bbecc83ebed4-kube-api-access-9vf9g\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:10.708616 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.708506    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:10.708616 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.708520    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd5a458-625c-4eec-b0df-bbecc83ebed4-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:10.708616 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.708533    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqz2c\" (UniqueName: \"kubernetes.io/projected/984aa715-0b1d-450d-a28c-cc85e4beb848-kube-api-access-dqz2c\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:10.801721 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.801678    6099 generic.go:356] "Generic (PLEG): container finished" podID="2bd5a458-625c-4eec-b0df-bbecc83ebed4" containerID="564bb0e684e51330212851e56a666ce6eeee2fb176b3384586b9bf7015a41ff8" exitCode=0
Oct 30 00:59:10.801907 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.801770    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l" event={"ID":"2bd5a458-625c-4eec-b0df-bbecc83ebed4","Type":"ContainerDied","Data":"564bb0e684e51330212851e56a666ce6eeee2fb176b3384586b9bf7015a41ff8"}
Oct 30 00:59:10.801907 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.801830    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l" event={"ID":"2bd5a458-625c-4eec-b0df-bbecc83ebed4","Type":"ContainerDied","Data":"29adb050b819d800768ced889270438e78e334d295355de5f30fa77f1982d0fb"}
Oct 30 00:59:10.801907 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.801779    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"
Oct 30 00:59:10.801907 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.801861    6099 scope.go:122] "RemoveContainer" containerID="564bb0e684e51330212851e56a666ce6eeee2fb176b3384586b9bf7015a41ff8"
Oct 30 00:59:10.803756 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.803685    6099 generic.go:356] "Generic (PLEG): container finished" podID="d6a88bf7-5dfd-4a6b-827a-4dbad7d58828" containerID="149e182606a19627d05ec99ce7af8a8987fd988d7cbec4916c21c92839b2d87f" exitCode=0
Oct 30 00:59:10.803988 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.803761    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx" event={"ID":"d6a88bf7-5dfd-4a6b-827a-4dbad7d58828","Type":"ContainerDied","Data":"149e182606a19627d05ec99ce7af8a8987fd988d7cbec4916c21c92839b2d87f"}
Oct 30 00:59:10.804114 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.804010    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx" event={"ID":"d6a88bf7-5dfd-4a6b-827a-4dbad7d58828","Type":"ContainerDied","Data":"56dd927a802271426b2f7418b5f5ef98a89991b023a173baef05aee793139726"}
Oct 30 00:59:10.804114 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.803820    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx"
Oct 30 00:59:10.805882 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.805857    6099 generic.go:356] "Generic (PLEG): container finished" podID="984aa715-0b1d-450d-a28c-cc85e4beb848" containerID="b5ea418f68adf5148147398c6ff891c14c4fbd0c1691cc658a9359bf42b43881" exitCode=0
Oct 30 00:59:10.806003 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.805886    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt"
Oct 30 00:59:10.806003 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.805939    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt" event={"ID":"984aa715-0b1d-450d-a28c-cc85e4beb848","Type":"ContainerDied","Data":"b5ea418f68adf5148147398c6ff891c14c4fbd0c1691cc658a9359bf42b43881"}
Oct 30 00:59:10.806003 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.805981    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt" event={"ID":"984aa715-0b1d-450d-a28c-cc85e4beb848","Type":"ContainerDied","Data":"c4862e94d20f152ff02f6111f0b031cc597de02a55a353b2caf01cbff14c4e98"}
Oct 30 00:59:10.817428 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.817402    6099 scope.go:122] "RemoveContainer" containerID="564bb0e684e51330212851e56a666ce6eeee2fb176b3384586b9bf7015a41ff8"
Oct 30 00:59:10.817837 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:59:10.817802    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"564bb0e684e51330212851e56a666ce6eeee2fb176b3384586b9bf7015a41ff8\": container with ID starting with 564bb0e684e51330212851e56a666ce6eeee2fb176b3384586b9bf7015a41ff8 not found: ID does not exist" containerID="564bb0e684e51330212851e56a666ce6eeee2fb176b3384586b9bf7015a41ff8"
Oct 30 00:59:10.817949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.817850    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"564bb0e684e51330212851e56a666ce6eeee2fb176b3384586b9bf7015a41ff8"} err="failed to get container status \"564bb0e684e51330212851e56a666ce6eeee2fb176b3384586b9bf7015a41ff8\": rpc error: code = NotFound desc = could not find container \"564bb0e684e51330212851e56a666ce6eeee2fb176b3384586b9bf7015a41ff8\": container with ID starting with 564bb0e684e51330212851e56a666ce6eeee2fb176b3384586b9bf7015a41ff8 not found: ID does not exist"
Oct 30 00:59:10.817949 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.817875    6099 scope.go:122] "RemoveContainer" containerID="149e182606a19627d05ec99ce7af8a8987fd988d7cbec4916c21c92839b2d87f"
Oct 30 00:59:10.830501 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.830480    6099 scope.go:122] "RemoveContainer" containerID="149e182606a19627d05ec99ce7af8a8987fd988d7cbec4916c21c92839b2d87f"
Oct 30 00:59:10.830839 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:59:10.830814    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"149e182606a19627d05ec99ce7af8a8987fd988d7cbec4916c21c92839b2d87f\": container with ID starting with 149e182606a19627d05ec99ce7af8a8987fd988d7cbec4916c21c92839b2d87f not found: ID does not exist" containerID="149e182606a19627d05ec99ce7af8a8987fd988d7cbec4916c21c92839b2d87f"
Oct 30 00:59:10.830942 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.830845    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149e182606a19627d05ec99ce7af8a8987fd988d7cbec4916c21c92839b2d87f"} err="failed to get container status \"149e182606a19627d05ec99ce7af8a8987fd988d7cbec4916c21c92839b2d87f\": rpc error: code = NotFound desc = could not find container \"149e182606a19627d05ec99ce7af8a8987fd988d7cbec4916c21c92839b2d87f\": container with ID starting with 149e182606a19627d05ec99ce7af8a8987fd988d7cbec4916c21c92839b2d87f not found: ID does not exist"
Oct 30 00:59:10.830942 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.830866    6099 scope.go:122] "RemoveContainer" containerID="b5ea418f68adf5148147398c6ff891c14c4fbd0c1691cc658a9359bf42b43881"
Oct 30 00:59:10.845269 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.845248    6099 scope.go:122] "RemoveContainer" containerID="b5ea418f68adf5148147398c6ff891c14c4fbd0c1691cc658a9359bf42b43881"
Oct 30 00:59:10.846023 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:59:10.845993    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ea418f68adf5148147398c6ff891c14c4fbd0c1691cc658a9359bf42b43881\": container with ID starting with b5ea418f68adf5148147398c6ff891c14c4fbd0c1691cc658a9359bf42b43881 not found: ID does not exist" containerID="b5ea418f68adf5148147398c6ff891c14c4fbd0c1691cc658a9359bf42b43881"
Oct 30 00:59:10.846132 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.846032    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ea418f68adf5148147398c6ff891c14c4fbd0c1691cc658a9359bf42b43881"} err="failed to get container status \"b5ea418f68adf5148147398c6ff891c14c4fbd0c1691cc658a9359bf42b43881\": rpc error: code = NotFound desc = could not find container \"b5ea418f68adf5148147398c6ff891c14c4fbd0c1691cc658a9359bf42b43881\": container with ID starting with b5ea418f68adf5148147398c6ff891c14c4fbd0c1691cc658a9359bf42b43881 not found: ID does not exist"
Oct 30 00:59:10.871409 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.871383    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"]
Oct 30 00:59:10.875790 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.875754    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-82f6l"]
Oct 30 00:59:10.920801 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.920775    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx"]
Oct 30 00:59:10.925933 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.925912    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-353403-75df964455-rttsx"]
Oct 30 00:59:10.959101 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.959066    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt"]
Oct 30 00:59:10.963111 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:10.963090    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-353403-55845f89db-zx8nt"]
Oct 30 00:59:12.264386 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:12.264354    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd5a458-625c-4eec-b0df-bbecc83ebed4" path="/var/lib/kubelet/pods/2bd5a458-625c-4eec-b0df-bbecc83ebed4/volumes"
Oct 30 00:59:12.264898 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:12.264883    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="984aa715-0b1d-450d-a28c-cc85e4beb848" path="/var/lib/kubelet/pods/984aa715-0b1d-450d-a28c-cc85e4beb848/volumes"
Oct 30 00:59:12.265320 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:12.265305    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6a88bf7-5dfd-4a6b-827a-4dbad7d58828" path="/var/lib/kubelet/pods/d6a88bf7-5dfd-4a6b-827a-4dbad7d58828/volumes"
Oct 30 00:59:16.631843 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:59:16.631804    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:59:19.889439 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:19.889362    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt"]
Oct 30 00:59:19.900721 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:19.900684    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a4fe0143-bbjrt"]
Oct 30 00:59:20.264169 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:20.264089    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b852a375-9dcb-4094-88b0-dca00f669314" path="/var/lib/kubelet/pods/b852a375-9dcb-4094-88b0-dca00f669314/volumes"
Oct 30 00:59:20.798224 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:20.798191    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"]
Oct 30 00:59:20.798417 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:20.798398    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9" podUID="e8270463-2deb-44c8-baa7-9ccfa2167945" containerName="istio-proxy" containerID="cri-o://0bef5f1ae66940b0f1fd5b45bc0f5da8d4b6b3fb1e2c7c78558e52f9e94262b6" gracePeriod=30
Oct 30 00:59:20.810629 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:20.810579    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"]
Oct 30 00:59:20.810879 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:20.810849    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c" podUID="29d2f6eb-bea1-4aec-baaa-4146db3218e3" containerName="istio-proxy" containerID="cri-o://9582410bc1d63b2aff5b1ac9e4c3bc1cb9d47715b74e2d0cf56f124d316d4194" gracePeriod=30
Oct 30 00:59:20.821505 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:20.821479    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"]
Oct 30 00:59:20.821754 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:20.821731    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt" podUID="6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" containerName="istio-proxy" containerID="cri-o://f8794b2694b56f565efe6dbc0dffa0c03d80ba4c86fcd84e2ebaf599adc5eab3" gracePeriod=30
Oct 30 00:59:26.134140 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.134096    6099 generic.go:356] "Generic (PLEG): container finished" podID="6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" containerID="f8794b2694b56f565efe6dbc0dffa0c03d80ba4c86fcd84e2ebaf599adc5eab3" exitCode=0
Oct 30 00:59:26.134655 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.134284    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt" event={"ID":"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0","Type":"ContainerDied","Data":"f8794b2694b56f565efe6dbc0dffa0c03d80ba4c86fcd84e2ebaf599adc5eab3"}
Oct 30 00:59:26.138281 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.138035    6099 generic.go:356] "Generic (PLEG): container finished" podID="29d2f6eb-bea1-4aec-baaa-4146db3218e3" containerID="9582410bc1d63b2aff5b1ac9e4c3bc1cb9d47715b74e2d0cf56f124d316d4194" exitCode=0
Oct 30 00:59:26.138281 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.138172    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c" event={"ID":"29d2f6eb-bea1-4aec-baaa-4146db3218e3","Type":"ContainerDied","Data":"9582410bc1d63b2aff5b1ac9e4c3bc1cb9d47715b74e2d0cf56f124d316d4194"}
Oct 30 00:59:26.139985 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.139960    6099 generic.go:356] "Generic (PLEG): container finished" podID="e8270463-2deb-44c8-baa7-9ccfa2167945" containerID="0bef5f1ae66940b0f1fd5b45bc0f5da8d4b6b3fb1e2c7c78558e52f9e94262b6" exitCode=0
Oct 30 00:59:26.140093 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.140034    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9" event={"ID":"e8270463-2deb-44c8-baa7-9ccfa2167945","Type":"ContainerDied","Data":"0bef5f1ae66940b0f1fd5b45bc0f5da8d4b6b3fb1e2c7c78558e52f9e94262b6"}
Oct 30 00:59:26.195334 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.195305    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:59:26.229327 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.229292    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:59:26.246149 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.246125    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:59:26.350424 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.350337    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d2f6eb-bea1-4aec-baaa-4146db3218e3-kube-api-access-9kz7n\" (UniqueName: \"kubernetes.io/projected/29d2f6eb-bea1-4aec-baaa-4146db3218e3-kube-api-access-9kz7n\") pod \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") "
Oct 30 00:59:26.350424 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.350407    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-workload-socket\") pod \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") "
Oct 30 00:59:26.350666 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.350521    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-token\" (UniqueName: \"kubernetes.io/projected/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-token\") pod \"e8270463-2deb-44c8-baa7-9ccfa2167945\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") "
Oct 30 00:59:26.350666 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.350563    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istiod-ca-cert\") pod \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") "
Oct 30 00:59:26.350666 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.350640    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8270463-2deb-44c8-baa7-9ccfa2167945-kube-api-access-49bd9\" (UniqueName: \"kubernetes.io/projected/e8270463-2deb-44c8-baa7-9ccfa2167945-kube-api-access-49bd9\") pod \"e8270463-2deb-44c8-baa7-9ccfa2167945\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") "
Oct 30 00:59:26.350927 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.350689    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-envoy\") pod \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") "
Oct 30 00:59:26.350927 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.350764    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-podinfo\") pod \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") "
Oct 30 00:59:26.350927 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.350812    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-token\" (UniqueName: \"kubernetes.io/projected/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-token\") pod \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") "
Oct 30 00:59:26.350927 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.350867    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-credential-socket\") pod \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") "
Oct 30 00:59:26.350927 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.350912    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-envoy\") pod \"e8270463-2deb-44c8-baa7-9ccfa2167945\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") "
Oct 30 00:59:26.351186 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.350951    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-credential-socket\") pod \"e8270463-2deb-44c8-baa7-9ccfa2167945\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") "
Oct 30 00:59:26.351186 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.351043    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-workload-socket\") pod \"e8270463-2deb-44c8-baa7-9ccfa2167945\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") "
Oct 30 00:59:26.351186 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.351092    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-data\") pod \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") "
Oct 30 00:59:26.351186 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.351129    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-workload-certs\") pod \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") "
Oct 30 00:59:26.351186 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.351164    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-workload-certs\") pod \"e8270463-2deb-44c8-baa7-9ccfa2167945\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") "
Oct 30 00:59:26.351482 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.351209    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-data\") pod \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") "
Oct 30 00:59:26.351482 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.351240    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8270463-2deb-44c8-baa7-9ccfa2167945-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8270463-2deb-44c8-baa7-9ccfa2167945-istiod-ca-cert\") pod \"e8270463-2deb-44c8-baa7-9ccfa2167945\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") "
Oct 30 00:59:26.351482 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.351282    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-podinfo\") pod \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") "
Oct 30 00:59:26.351482 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.351322    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-workload-socket\") pod \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") "
Oct 30 00:59:26.351482 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.351371    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-envoy\") pod \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") "
Oct 30 00:59:26.351482 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.351442    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-token\" (UniqueName: \"kubernetes.io/projected/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-token\") pod \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") "
Oct 30 00:59:26.351482 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.351478    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-data\") pod \"e8270463-2deb-44c8-baa7-9ccfa2167945\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") "
Oct 30 00:59:26.351869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.351525    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-kube-api-access-hc2z6\" (UniqueName: \"kubernetes.io/projected/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-kube-api-access-hc2z6\") pod \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") "
Oct 30 00:59:26.351869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.351568    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-workload-certs\") pod \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") "
Oct 30 00:59:26.351869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.351666    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istiod-ca-cert" pod "29d2f6eb-bea1-4aec-baaa-4146db3218e3" (UID: "29d2f6eb-bea1-4aec-baaa-4146db3218e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:26.351869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.351688    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-credential-socket\") pod \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\" (UID: \"29d2f6eb-bea1-4aec-baaa-4146db3218e3\") "
Oct 30 00:59:26.351869 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.351785    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-podinfo\") pod \"e8270463-2deb-44c8-baa7-9ccfa2167945\" (UID: \"e8270463-2deb-44c8-baa7-9ccfa2167945\") "
Oct 30 00:59:26.352423 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.351889    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istiod-ca-cert\") pod \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\" (UID: \"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0\") "
Oct 30 00:59:26.352423 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.351982    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-credential-socket" pod "e8270463-2deb-44c8-baa7-9ccfa2167945" (UID: "e8270463-2deb-44c8-baa7-9ccfa2167945"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:26.352423 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.352266    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-credential-socket" pod "6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" (UID: "6ee91a08-3dfc-47e1-bac6-bbac9da01bc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:26.353105 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.353079    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-workload-socket" pod "6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" (UID: "6ee91a08-3dfc-47e1-bac6-bbac9da01bc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:26.353500 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.353478    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.353666 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.353650    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.353790 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.353775    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.353927 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.353913    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.354020 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.353654    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-workload-socket" pod "e8270463-2deb-44c8-baa7-9ccfa2167945" (UID: "e8270463-2deb-44c8-baa7-9ccfa2167945"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:26.354095 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.353846    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-data" pod "6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" (UID: "6ee91a08-3dfc-47e1-bac6-bbac9da01bc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:26.354162 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.354022    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-workload-certs" pod "6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" (UID: "6ee91a08-3dfc-47e1-bac6-bbac9da01bc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:26.354228 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.354187    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-workload-certs" pod "e8270463-2deb-44c8-baa7-9ccfa2167945" (UID: "e8270463-2deb-44c8-baa7-9ccfa2167945"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:26.354407 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.354384    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-data" pod "29d2f6eb-bea1-4aec-baaa-4146db3218e3" (UID: "29d2f6eb-bea1-4aec-baaa-4146db3218e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:26.354665 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.354640    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8270463-2deb-44c8-baa7-9ccfa2167945-istiod-ca-cert" pod "e8270463-2deb-44c8-baa7-9ccfa2167945" (UID: "e8270463-2deb-44c8-baa7-9ccfa2167945"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:26.354843 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.354733    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-token" pod "6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" (UID: "6ee91a08-3dfc-47e1-bac6-bbac9da01bc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:26.355647 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.355624    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istiod-ca-cert" pod "6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" (UID: "6ee91a08-3dfc-47e1-bac6-bbac9da01bc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:26.358165 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.355830    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-envoy" pod "29d2f6eb-bea1-4aec-baaa-4146db3218e3" (UID: "29d2f6eb-bea1-4aec-baaa-4146db3218e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:26.358314 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.357091    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-podinfo" pod "29d2f6eb-bea1-4aec-baaa-4146db3218e3" (UID: "29d2f6eb-bea1-4aec-baaa-4146db3218e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:26.358428 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.357160    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8270463-2deb-44c8-baa7-9ccfa2167945-kube-api-access-49bd9" pod "e8270463-2deb-44c8-baa7-9ccfa2167945" (UID: "e8270463-2deb-44c8-baa7-9ccfa2167945"). InnerVolumeSpecName "kube-api-access-49bd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:26.358539 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.357337    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-podinfo" pod "6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" (UID: "6ee91a08-3dfc-47e1-bac6-bbac9da01bc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:26.358652 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.357377    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d2f6eb-bea1-4aec-baaa-4146db3218e3-kube-api-access-9kz7n" pod "29d2f6eb-bea1-4aec-baaa-4146db3218e3" (UID: "29d2f6eb-bea1-4aec-baaa-4146db3218e3"). InnerVolumeSpecName "kube-api-access-9kz7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:26.358763 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.357674    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-data" pod "e8270463-2deb-44c8-baa7-9ccfa2167945" (UID: "e8270463-2deb-44c8-baa7-9ccfa2167945"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:26.358872 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.357890    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-workload-socket" pod "29d2f6eb-bea1-4aec-baaa-4146db3218e3" (UID: "29d2f6eb-bea1-4aec-baaa-4146db3218e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:26.359065 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.357939    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-workload-certs" pod "29d2f6eb-bea1-4aec-baaa-4146db3218e3" (UID: "29d2f6eb-bea1-4aec-baaa-4146db3218e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:26.359172 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.358089    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-credential-socket" pod "29d2f6eb-bea1-4aec-baaa-4146db3218e3" (UID: "29d2f6eb-bea1-4aec-baaa-4146db3218e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:26.359285 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.359018    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-token" pod "e8270463-2deb-44c8-baa7-9ccfa2167945" (UID: "e8270463-2deb-44c8-baa7-9ccfa2167945"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:26.359537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.359508    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-envoy" pod "e8270463-2deb-44c8-baa7-9ccfa2167945" (UID: "e8270463-2deb-44c8-baa7-9ccfa2167945"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:26.359923 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.359892    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-envoy" pod "6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" (UID: "6ee91a08-3dfc-47e1-bac6-bbac9da01bc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:26.360235 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.360207    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-kube-api-access-hc2z6" pod "6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" (UID: "6ee91a08-3dfc-47e1-bac6-bbac9da01bc0"). InnerVolumeSpecName "kube-api-access-hc2z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:26.360498 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.360478    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-token" pod "29d2f6eb-bea1-4aec-baaa-4146db3218e3" (UID: "29d2f6eb-bea1-4aec-baaa-4146db3218e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:26.361649 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.361626    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-podinfo" pod "e8270463-2deb-44c8-baa7-9ccfa2167945" (UID: "e8270463-2deb-44c8-baa7-9ccfa2167945"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:26.464324 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464290    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.464324 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464326    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.464537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464342    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.464537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464356    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.464537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464368    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8270463-2deb-44c8-baa7-9ccfa2167945-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.464537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464380    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.464537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464392    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.464537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464405    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.464537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464420    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.464537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464432    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.464537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464443    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc2z6\" (UniqueName: \"kubernetes.io/projected/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-kube-api-access-hc2z6\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.464537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464457    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.464537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464470    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.464537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464482    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.464537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464495    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.464537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464509    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kz7n\" (UniqueName: \"kubernetes.io/projected/29d2f6eb-bea1-4aec-baaa-4146db3218e3-kube-api-access-9kz7n\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.464537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464520    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.464537 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464533    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49bd9\" (UniqueName: \"kubernetes.io/projected/e8270463-2deb-44c8-baa7-9ccfa2167945-kube-api-access-49bd9\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.465068 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464545    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.465068 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464558    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d2f6eb-bea1-4aec-baaa-4146db3218e3-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.465068 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464569    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.465068 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464581    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:26.465068 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:26.464593    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8270463-2deb-44c8-baa7-9ccfa2167945-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:27.034737 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.034667    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj"]
Oct 30 00:59:27.038730 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.038696    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06521758-czhvj"]
Oct 30 00:59:27.152091 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.152008    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"
Oct 30 00:59:27.152091 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.152013    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt" event={"ID":"6ee91a08-3dfc-47e1-bac6-bbac9da01bc0","Type":"ContainerDied","Data":"b87a468dbb799b4edc002b52bb6dc538432bfb003f41f73c1982a9fa324678e5"}
Oct 30 00:59:27.152589 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.152148    6099 scope.go:122] "RemoveContainer" containerID="f8794b2694b56f565efe6dbc0dffa0c03d80ba4c86fcd84e2ebaf599adc5eab3"
Oct 30 00:59:27.153863 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.153823    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"
Oct 30 00:59:27.154041 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.153892    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c" event={"ID":"29d2f6eb-bea1-4aec-baaa-4146db3218e3","Type":"ContainerDied","Data":"b3d38fbee0fb93017a783d85735c63a4ede70f86980a61a9f7922ef85e7648e6"}
Oct 30 00:59:27.161216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.161190    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"
Oct 30 00:59:27.161342 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.161215    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9" event={"ID":"e8270463-2deb-44c8-baa7-9ccfa2167945","Type":"ContainerDied","Data":"c51c0c06714027817fdba5479b0a38e328ec2d733ddcaa0aedd557ce14feba36"}
Oct 30 00:59:27.191833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.191800    6099 scope.go:122] "RemoveContainer" containerID="9582410bc1d63b2aff5b1ac9e4c3bc1cb9d47715b74e2d0cf56f124d316d4194"
Oct 30 00:59:27.204233 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.204206    6099 scope.go:122] "RemoveContainer" containerID="0bef5f1ae66940b0f1fd5b45bc0f5da8d4b6b3fb1e2c7c78558e52f9e94262b6"
Oct 30 00:59:27.209765 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.209740    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"]
Oct 30 00:59:27.220901 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.220878    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-lcrlt"]
Oct 30 00:59:27.264872 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.264843    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"]
Oct 30 00:59:27.268808 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.268786    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-wlp9c"]
Oct 30 00:59:27.306121 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.306096    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"]
Oct 30 00:59:27.311094 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.311069    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-z5kr9"]
Oct 30 00:59:27.353448 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.353406    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"]
Oct 30 00:59:27.353671 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.353638    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv" podUID="99e50152-55fb-4e62-a4af-0efbb35fc8df" containerName="istio-proxy" containerID="cri-o://91a62bc9e59c814f5393173a63c87772d25d24af64291d5ff43c81d486a29b0f" gracePeriod=30
Oct 30 00:59:27.359322 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.359291    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"]
Oct 30 00:59:27.359564 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.359533    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx" podUID="e764760a-aad7-44be-9068-387287c0602c" containerName="istio-proxy" containerID="cri-o://d9fac4f2d3efb6e291959cba6511e86001b2117fba7e6efc6f7452822f6c4dcd" gracePeriod=30
Oct 30 00:59:27.365903 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.365877    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"]
Oct 30 00:59:27.366171 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.366136    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd" podUID="03d0a6b5-42e2-4180-84f2-9bd6a686da0e" containerName="istio-proxy" containerID="cri-o://d3b278a654e7d3f80a3faf005e2a447d9b2bbe969694b0c33579f7bfc931a7b0" gracePeriod=30
Oct 30 00:59:27.371925 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.371904    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"]
Oct 30 00:59:27.372128 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.372105    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm" podUID="75118b09-18bb-4678-b966-5df63a1720bf" containerName="istio-proxy" containerID="cri-o://8d06aa87ba2dd99a82f43b6f6f8f45a732298415973e5ec73cc89e7193721860" gracePeriod=30
Oct 30 00:59:27.376976 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.376950    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"]
Oct 30 00:59:27.377175 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.377154    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5" podUID="c030a78c-6e98-43a0-927b-a4caee58d9fa" containerName="istio-proxy" containerID="cri-o://344c718722a9b7d9c4ba28ef84052530801ecec077961775300540c5df0d73e1" gracePeriod=30
Oct 30 00:59:27.405434 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.405354    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"]
Oct 30 00:59:27.405626 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.405600    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp" podUID="cd226360-69bc-45d8-95f6-9e2d00e9e4ce" containerName="istio-proxy" containerID="cri-o://3568018eeae415a06f67b2380ae899c9d32dbedc4a95fc1dd9272b44dc12cbe8" gracePeriod=30
Oct 30 00:59:27.409216 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.409193    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"]
Oct 30 00:59:27.409378 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:27.409358    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj" podUID="1638dd27-e6e3-49e0-83cd-95e91893aa1c" containerName="istio-proxy" containerID="cri-o://2e496e88c9af041b98e19ab627d12407f0d703d70850ab109fdf8c69a13b63d8" gracePeriod=30
Oct 30 00:59:28.265152 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.265122    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29d2f6eb-bea1-4aec-baaa-4146db3218e3" path="/var/lib/kubelet/pods/29d2f6eb-bea1-4aec-baaa-4146db3218e3/volumes"
Oct 30 00:59:28.265766 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.265748    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d33f44-8a31-4aa9-b169-3d1f819a0335" path="/var/lib/kubelet/pods/63d33f44-8a31-4aa9-b169-3d1f819a0335/volumes"
Oct 30 00:59:28.266266 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.266252    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" path="/var/lib/kubelet/pods/6ee91a08-3dfc-47e1-bac6-bbac9da01bc0/volumes"
Oct 30 00:59:28.267114 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.267099    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8270463-2deb-44c8-baa7-9ccfa2167945" path="/var/lib/kubelet/pods/e8270463-2deb-44c8-baa7-9ccfa2167945/volumes"
Oct 30 00:59:28.782727 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.760939    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc"]
Oct 30 00:59:28.789141 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.789104    6099 cpu_manager.go:414] "containerMap: removing container" podUID="d6a88bf7-5dfd-4a6b-827a-4dbad7d58828" containerName="payload-pre-processing"
Oct 30 00:59:28.789340 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.789321    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6a88bf7-5dfd-4a6b-827a-4dbad7d58828" containerName="payload-pre-processing"
Oct 30 00:59:28.789481 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.789468    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5ec09452-0400-491f-b1df-65d1e4160023" containerName="payload-pre-processing"
Oct 30 00:59:28.789559 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.789548    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec09452-0400-491f-b1df-65d1e4160023" containerName="payload-pre-processing"
Oct 30 00:59:28.789659 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.789650    6099 cpu_manager.go:414] "containerMap: removing container" podUID="2bd5a458-625c-4eec-b0df-bbecc83ebed4" containerName="istio-proxy"
Oct 30 00:59:28.789752 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.789739    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd5a458-625c-4eec-b0df-bbecc83ebed4" containerName="istio-proxy"
Oct 30 00:59:28.789839 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.789830    6099 cpu_manager.go:414] "containerMap: removing container" podUID="b852a375-9dcb-4094-88b0-dca00f669314" containerName="revoke-keys"
Oct 30 00:59:28.789946 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.789897    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b852a375-9dcb-4094-88b0-dca00f669314" containerName="revoke-keys"
Oct 30 00:59:28.790071 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.790059    6099 cpu_manager.go:414] "containerMap: removing container" podUID="984aa715-0b1d-450d-a28c-cc85e4beb848" containerName="payload-processing"
Oct 30 00:59:28.790147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.790138    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="984aa715-0b1d-450d-a28c-cc85e4beb848" containerName="payload-processing"
Oct 30 00:59:28.790244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.790236    6099 cpu_manager.go:414] "containerMap: removing container" podUID="29d2f6eb-bea1-4aec-baaa-4146db3218e3" containerName="istio-proxy"
Oct 30 00:59:28.790311 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.790302    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d2f6eb-bea1-4aec-baaa-4146db3218e3" containerName="istio-proxy"
Oct 30 00:59:28.790380 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.790372    6099 cpu_manager.go:414] "containerMap: removing container" podUID="6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" containerName="istio-proxy"
Oct 30 00:59:28.790448 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.790439    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" containerName="istio-proxy"
Oct 30 00:59:28.790540 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.790531    6099 cpu_manager.go:414] "containerMap: removing container" podUID="42140e21-60da-460b-8d55-d7b9f5e53a2d" containerName="payload-processing"
Oct 30 00:59:28.790612 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.790602    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42140e21-60da-460b-8d55-d7b9f5e53a2d" containerName="payload-processing"
Oct 30 00:59:28.790765 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.790720    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e8270463-2deb-44c8-baa7-9ccfa2167945" containerName="istio-proxy"
Oct 30 00:59:28.790885 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.790869    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8270463-2deb-44c8-baa7-9ccfa2167945" containerName="istio-proxy"
Oct 30 00:59:28.791191 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.791175    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8270463-2deb-44c8-baa7-9ccfa2167945" containerName="istio-proxy"
Oct 30 00:59:28.791301 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.791291    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6a88bf7-5dfd-4a6b-827a-4dbad7d58828" containerName="payload-pre-processing"
Oct 30 00:59:28.791378 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.791368    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec09452-0400-491f-b1df-65d1e4160023" containerName="payload-pre-processing"
Oct 30 00:59:28.791453 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.791444    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd5a458-625c-4eec-b0df-bbecc83ebed4" containerName="istio-proxy"
Oct 30 00:59:28.791528 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.791519    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="b852a375-9dcb-4094-88b0-dca00f669314" containerName="revoke-keys"
Oct 30 00:59:28.791613 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.791602    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="984aa715-0b1d-450d-a28c-cc85e4beb848" containerName="payload-processing"
Oct 30 00:59:28.791729 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.791695    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d2f6eb-bea1-4aec-baaa-4146db3218e3" containerName="istio-proxy"
Oct 30 00:59:28.791831 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.791819    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee91a08-3dfc-47e1-bac6-bbac9da01bc0" containerName="istio-proxy"
Oct 30 00:59:28.791922 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.791912    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="42140e21-60da-460b-8d55-d7b9f5e53a2d" containerName="payload-processing"
Oct 30 00:59:28.800040 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.800011    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc"]
Oct 30 00:59:28.800265 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.800122    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc"
Oct 30 00:59:28.836190 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.828261    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:59:28.836190 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.828355    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:59:28.936526 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.936485    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55372fd8-ea01-4397-be34-ab2260d9ef33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8090513-mdxqc\" (UID: \"55372fd8-ea01-4397-be34-ab2260d9ef33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc"
Oct 30 00:59:28.936758 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.936726    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6vg6\" (UniqueName: \"kubernetes.io/projected/55372fd8-ea01-4397-be34-ab2260d9ef33-kube-api-access-t6vg6\") pod \"maas-api-revoke-keys-e2e-mt-d8090513-mdxqc\" (UID: \"55372fd8-ea01-4397-be34-ab2260d9ef33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc"
Oct 30 00:59:28.936881 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:28.936862    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55372fd8-ea01-4397-be34-ab2260d9ef33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8090513-mdxqc\" (UID: \"55372fd8-ea01-4397-be34-ab2260d9ef33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc"
Oct 30 00:59:29.039064 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:29.039031    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6vg6\" (UniqueName: \"kubernetes.io/projected/55372fd8-ea01-4397-be34-ab2260d9ef33-kube-api-access-t6vg6\") pod \"maas-api-revoke-keys-e2e-mt-d8090513-mdxqc\" (UID: \"55372fd8-ea01-4397-be34-ab2260d9ef33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc"
Oct 30 00:59:29.039272 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:29.039175    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55372fd8-ea01-4397-be34-ab2260d9ef33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8090513-mdxqc\" (UID: \"55372fd8-ea01-4397-be34-ab2260d9ef33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc"
Oct 30 00:59:29.039406 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:29.039390    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55372fd8-ea01-4397-be34-ab2260d9ef33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8090513-mdxqc\" (UID: \"55372fd8-ea01-4397-be34-ab2260d9ef33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc"
Oct 30 00:59:29.040002 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:29.039967    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55372fd8-ea01-4397-be34-ab2260d9ef33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8090513-mdxqc\" (UID: \"55372fd8-ea01-4397-be34-ab2260d9ef33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc"
Oct 30 00:59:29.041538 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:29.041516    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55372fd8-ea01-4397-be34-ab2260d9ef33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8090513-mdxqc\" (UID: \"55372fd8-ea01-4397-be34-ab2260d9ef33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc"
Oct 30 00:59:29.067791 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:29.067759    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6vg6\" (UniqueName: \"kubernetes.io/projected/55372fd8-ea01-4397-be34-ab2260d9ef33-kube-api-access-t6vg6\") pod \"maas-api-revoke-keys-e2e-mt-d8090513-mdxqc\" (UID: \"55372fd8-ea01-4397-be34-ab2260d9ef33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc"
Oct 30 00:59:29.151859 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:29.151783    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc"
Oct 30 00:59:29.516982 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:59:29.516949    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55372fd8_ea01_4397_be34_ab2260d9ef33.slice/crio-b71a87a0aa36ee7ae8887740d836e2d0968a343ac24effc4f01ced0847341bd2 WatchSource:0}: Error finding container b71a87a0aa36ee7ae8887740d836e2d0968a343ac24effc4f01ced0847341bd2: Status 404 returned error can't find the container with id b71a87a0aa36ee7ae8887740d836e2d0968a343ac24effc4f01ced0847341bd2
Oct 30 00:59:29.517311 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:29.517037    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc"]
Oct 30 00:59:30.241040 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:30.241007    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc" event={"ID":"55372fd8-ea01-4397-be34-ab2260d9ef33","Type":"ContainerStarted","Data":"d81c7c210ad018cbdbb899188b2f9e1dedd47360ac4abd32c85c1a05816b46cd"}
Oct 30 00:59:30.241207 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:30.241065    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc" event={"ID":"55372fd8-ea01-4397-be34-ab2260d9ef33","Type":"ContainerStarted","Data":"b71a87a0aa36ee7ae8887740d836e2d0968a343ac24effc4f01ced0847341bd2"}
Oct 30 00:59:30.283587 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:30.283538    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc" podStartSLOduration=2.283524692 podStartE2EDuration="2.283524692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:30.282804875 +0000 UTC m=+1464.139712631" watchObservedRunningTime="2026-10-30 00:59:30.283524692 +0000 UTC m=+1464.140432498"
Oct 30 00:59:30.621130 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:30.621082    6099 patch_prober.go:29] interesting pod/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:59:30.621533 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:30.621168    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx" podUID="e764760a-aad7-44be-9068-387287c0602c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:59:31.253418 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:31.253380    6099 generic.go:356] "Generic (PLEG): container finished" podID="55372fd8-ea01-4397-be34-ab2260d9ef33" containerID="d81c7c210ad018cbdbb899188b2f9e1dedd47360ac4abd32c85c1a05816b46cd" exitCode=0
Oct 30 00:59:31.253625 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:31.253480    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc" event={"ID":"55372fd8-ea01-4397-be34-ab2260d9ef33","Type":"ContainerDied","Data":"d81c7c210ad018cbdbb899188b2f9e1dedd47360ac4abd32c85c1a05816b46cd"}
Oct 30 00:59:32.849773 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:32.846877    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc"
Oct 30 00:59:32.958055 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:32.958031    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:59:32.960254 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:32.960230    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:59:32.967588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:32.967567    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:59:32.982517 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:32.982491    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:59:32.992055 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:32.992029    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:59:32.996028 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:32.996010    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:59:32.999486 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:32.999465    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:59:33.020663 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.020632    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55372fd8-ea01-4397-be34-ab2260d9ef33-kube-api-access-t6vg6\" (UniqueName: \"kubernetes.io/projected/55372fd8-ea01-4397-be34-ab2260d9ef33-kube-api-access-t6vg6\") pod \"55372fd8-ea01-4397-be34-ab2260d9ef33\" (UID: \"55372fd8-ea01-4397-be34-ab2260d9ef33\") "
Oct 30 00:59:33.020964 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.020940    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55372fd8-ea01-4397-be34-ab2260d9ef33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55372fd8-ea01-4397-be34-ab2260d9ef33-maas-api-xks-ca\") pod \"55372fd8-ea01-4397-be34-ab2260d9ef33\" (UID: \"55372fd8-ea01-4397-be34-ab2260d9ef33\") "
Oct 30 00:59:33.021161 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.021141    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55372fd8-ea01-4397-be34-ab2260d9ef33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55372fd8-ea01-4397-be34-ab2260d9ef33-maas-api-openshift-ca\") pod \"55372fd8-ea01-4397-be34-ab2260d9ef33\" (UID: \"55372fd8-ea01-4397-be34-ab2260d9ef33\") "
Oct 30 00:59:33.021593 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.021565    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55372fd8-ea01-4397-be34-ab2260d9ef33-maas-api-openshift-ca" pod "55372fd8-ea01-4397-be34-ab2260d9ef33" (UID: "55372fd8-ea01-4397-be34-ab2260d9ef33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:33.022173 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.022148    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55372fd8-ea01-4397-be34-ab2260d9ef33-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.023230 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.023202    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55372fd8-ea01-4397-be34-ab2260d9ef33-maas-api-xks-ca" pod "55372fd8-ea01-4397-be34-ab2260d9ef33" (UID: "55372fd8-ea01-4397-be34-ab2260d9ef33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:59:33.023322 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.023263    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55372fd8-ea01-4397-be34-ab2260d9ef33-kube-api-access-t6vg6" pod "55372fd8-ea01-4397-be34-ab2260d9ef33" (UID: "55372fd8-ea01-4397-be34-ab2260d9ef33"). InnerVolumeSpecName "kube-api-access-t6vg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:33.123190 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123157    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-credential-socket\") pod \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") "
Oct 30 00:59:33.123389 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123198    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-workload-socket\") pod \"99e50152-55fb-4e62-a4af-0efbb35fc8df\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") "
Oct 30 00:59:33.123389 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123227    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-data\") pod \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") "
Oct 30 00:59:33.123389 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123252    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-credential-socket\") pod \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") "
Oct 30 00:59:33.123389 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123282    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-workload-socket\") pod \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") "
Oct 30 00:59:33.123389 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123313    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-data\") pod \"c030a78c-6e98-43a0-927b-a4caee58d9fa\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") "
Oct 30 00:59:33.123389 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123350    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e764760a-aad7-44be-9068-387287c0602c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e764760a-aad7-44be-9068-387287c0602c-istio-podinfo\") pod \"e764760a-aad7-44be-9068-387287c0602c\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") "
Oct 30 00:59:33.123389 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123376    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-podinfo\") pod \"c030a78c-6e98-43a0-927b-a4caee58d9fa\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") "
Oct 30 00:59:33.123810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123411    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-workload-certs\") pod \"75118b09-18bb-4678-b966-5df63a1720bf\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") "
Oct 30 00:59:33.123810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123482    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-workload-socket\") pod \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") "
Oct 30 00:59:33.123810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123515    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-envoy\") pod \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") "
Oct 30 00:59:33.123810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123517    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-credential-socket" pod "1638dd27-e6e3-49e0-83cd-95e91893aa1c" (UID: "1638dd27-e6e3-49e0-83cd-95e91893aa1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.123810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123529    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-workload-socket" pod "99e50152-55fb-4e62-a4af-0efbb35fc8df" (UID: "99e50152-55fb-4e62-a4af-0efbb35fc8df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.123810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123537    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-data" pod "cd226360-69bc-45d8-95f6-9e2d00e9e4ce" (UID: "cd226360-69bc-45d8-95f6-9e2d00e9e4ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.123810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123523    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-credential-socket" pod "cd226360-69bc-45d8-95f6-9e2d00e9e4ce" (UID: "cd226360-69bc-45d8-95f6-9e2d00e9e4ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.123810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123553    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-workload-certs\") pod \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") "
Oct 30 00:59:33.123810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123587    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istiod-ca-cert\") pod \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") "
Oct 30 00:59:33.123810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123622    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-token\" (UniqueName: \"kubernetes.io/projected/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-token\") pod \"99e50152-55fb-4e62-a4af-0efbb35fc8df\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") "
Oct 30 00:59:33.123810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123630    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-workload-socket" pod "1638dd27-e6e3-49e0-83cd-95e91893aa1c" (UID: "1638dd27-e6e3-49e0-83cd-95e91893aa1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.123810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123663    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-istio-data\") pod \"75118b09-18bb-4678-b966-5df63a1720bf\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") "
Oct 30 00:59:33.123810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123759    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-podinfo\") pod \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") "
Oct 30 00:59:33.123810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123788    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-envoy\") pod \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") "
Oct 30 00:59:33.123810 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123812    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-credential-socket\") pod \"99e50152-55fb-4e62-a4af-0efbb35fc8df\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123817    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-data" pod "c030a78c-6e98-43a0-927b-a4caee58d9fa" (UID: "c030a78c-6e98-43a0-927b-a4caee58d9fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123821    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-workload-socket" pod "03d0a6b5-42e2-4180-84f2-9bd6a686da0e" (UID: "03d0a6b5-42e2-4180-84f2-9bd6a686da0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123848    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-istio-envoy\") pod \"75118b09-18bb-4678-b966-5df63a1720bf\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123888    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-envoy\") pod \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123911    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-workload-certs" pod "03d0a6b5-42e2-4180-84f2-9bd6a686da0e" (UID: "03d0a6b5-42e2-4180-84f2-9bd6a686da0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123924    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-workload-socket\") pod \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.123967    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-token\" (UniqueName: \"kubernetes.io/projected/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-token\") pod \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.124001    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-workload-socket\") pod \"75118b09-18bb-4678-b966-5df63a1720bf\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.124019    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-workload-certs" pod "75118b09-18bb-4678-b966-5df63a1720bf" (UID: "75118b09-18bb-4678-b966-5df63a1720bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.124040    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75118b09-18bb-4678-b966-5df63a1720bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75118b09-18bb-4678-b966-5df63a1720bf-istio-podinfo\") pod \"75118b09-18bb-4678-b966-5df63a1720bf\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.124078    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e764760a-aad7-44be-9068-387287c0602c-kube-api-access-rxbh6\" (UniqueName: \"kubernetes.io/projected/e764760a-aad7-44be-9068-387287c0602c-kube-api-access-rxbh6\") pod \"e764760a-aad7-44be-9068-387287c0602c\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.124125    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-credential-socket\") pod \"75118b09-18bb-4678-b966-5df63a1720bf\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.124139    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-istio-data" pod "75118b09-18bb-4678-b966-5df63a1720bf" (UID: "75118b09-18bb-4678-b966-5df63a1720bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.124166    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c030a78c-6e98-43a0-927b-a4caee58d9fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c030a78c-6e98-43a0-927b-a4caee58d9fa-istiod-ca-cert\") pod \"c030a78c-6e98-43a0-927b-a4caee58d9fa\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.124158    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-credential-socket" pod "99e50152-55fb-4e62-a4af-0efbb35fc8df" (UID: "99e50152-55fb-4e62-a4af-0efbb35fc8df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.124196    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-workload-certs\") pod \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.124210    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-workload-socket" pod "cd226360-69bc-45d8-95f6-9e2d00e9e4ce" (UID: "cd226360-69bc-45d8-95f6-9e2d00e9e4ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.124411    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-workload-certs" pod "1638dd27-e6e3-49e0-83cd-95e91893aa1c" (UID: "1638dd27-e6e3-49e0-83cd-95e91893aa1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.124513    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istiod-ca-cert" pod "1638dd27-e6e3-49e0-83cd-95e91893aa1c" (UID: "1638dd27-e6e3-49e0-83cd-95e91893aa1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.126354    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-workload-certs\") pod \"e764760a-aad7-44be-9068-387287c0602c\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.126407    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-workload-socket\") pod \"e764760a-aad7-44be-9068-387287c0602c\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.126439    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-podinfo\") pod \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.126517    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istiod-ca-cert\") pod \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.126574    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-workload-socket\") pod \"c030a78c-6e98-43a0-927b-a4caee58d9fa\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.126609    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-workload-certs\") pod \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.126656    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-token\" (UniqueName: \"kubernetes.io/projected/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-token\") pod \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.126719    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-data\") pod \"99e50152-55fb-4e62-a4af-0efbb35fc8df\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.126756    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-data\") pod \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.126828    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75118b09-18bb-4678-b966-5df63a1720bf-kube-api-access-48cbl\" (UniqueName: \"kubernetes.io/projected/75118b09-18bb-4678-b966-5df63a1720bf-kube-api-access-48cbl\") pod \"75118b09-18bb-4678-b966-5df63a1720bf\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.126876    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-token\" (UniqueName: \"kubernetes.io/projected/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-token\") pod \"c030a78c-6e98-43a0-927b-a4caee58d9fa\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.126927    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-envoy\") pod \"99e50152-55fb-4e62-a4af-0efbb35fc8df\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.126988    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e764760a-aad7-44be-9068-387287c0602c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e764760a-aad7-44be-9068-387287c0602c-istiod-ca-cert\") pod \"e764760a-aad7-44be-9068-387287c0602c\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127032    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istiod-ca-cert\") pod \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127071    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99e50152-55fb-4e62-a4af-0efbb35fc8df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e50152-55fb-4e62-a4af-0efbb35fc8df-istiod-ca-cert\") pod \"99e50152-55fb-4e62-a4af-0efbb35fc8df\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127098    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-credential-socket\") pod \"c030a78c-6e98-43a0-927b-a4caee58d9fa\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127143    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-podinfo\") pod \"99e50152-55fb-4e62-a4af-0efbb35fc8df\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127173    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c030a78c-6e98-43a0-927b-a4caee58d9fa-kube-api-access-czm6t\" (UniqueName: \"kubernetes.io/projected/c030a78c-6e98-43a0-927b-a4caee58d9fa-kube-api-access-czm6t\") pod \"c030a78c-6e98-43a0-927b-a4caee58d9fa\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127224    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75118b09-18bb-4678-b966-5df63a1720bf-istio-token\" (UniqueName: \"kubernetes.io/projected/75118b09-18bb-4678-b966-5df63a1720bf-istio-token\") pod \"75118b09-18bb-4678-b966-5df63a1720bf\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127268    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-istio-envoy\") pod \"e764760a-aad7-44be-9068-387287c0602c\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127348    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-kube-api-access-w6wrw\" (UniqueName: \"kubernetes.io/projected/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-kube-api-access-w6wrw\") pod \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127396    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-workload-certs\") pod \"99e50152-55fb-4e62-a4af-0efbb35fc8df\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127455    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-kube-api-access-n67wh\" (UniqueName: \"kubernetes.io/projected/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-kube-api-access-n67wh\") pod \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127487    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e50152-55fb-4e62-a4af-0efbb35fc8df-kube-api-access-nt4f9\" (UniqueName: \"kubernetes.io/projected/99e50152-55fb-4e62-a4af-0efbb35fc8df-kube-api-access-nt4f9\") pod \"99e50152-55fb-4e62-a4af-0efbb35fc8df\" (UID: \"99e50152-55fb-4e62-a4af-0efbb35fc8df\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127500    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c030a78c-6e98-43a0-927b-a4caee58d9fa-istiod-ca-cert" pod "c030a78c-6e98-43a0-927b-a4caee58d9fa" (UID: "c030a78c-6e98-43a0-927b-a4caee58d9fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127536    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-token\" (UniqueName: \"kubernetes.io/projected/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-token\") pod \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\" (UID: \"cd226360-69bc-45d8-95f6-9e2d00e9e4ce\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127567    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-istio-data\") pod \"e764760a-aad7-44be-9068-387287c0602c\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127151    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-credential-socket" pod "75118b09-18bb-4678-b966-5df63a1720bf" (UID: "75118b09-18bb-4678-b966-5df63a1720bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127611    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-data\") pod \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127651    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e764760a-aad7-44be-9068-387287c0602c-istio-token\" (UniqueName: \"kubernetes.io/projected/e764760a-aad7-44be-9068-387287c0602c-istio-token\") pod \"e764760a-aad7-44be-9068-387287c0602c\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127686    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-envoy\") pod \"c030a78c-6e98-43a0-927b-a4caee58d9fa\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127731    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-podinfo\") pod \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") "
Oct 30 00:59:33.127743 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127786    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1638dd27-e6e3-49e0-83cd-95e91893aa1c-kube-api-access-wdz7b\" (UniqueName: \"kubernetes.io/projected/1638dd27-e6e3-49e0-83cd-95e91893aa1c-kube-api-access-wdz7b\") pod \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\" (UID: \"1638dd27-e6e3-49e0-83cd-95e91893aa1c\") "
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127823    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-credential-socket\") pod \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\" (UID: \"03d0a6b5-42e2-4180-84f2-9bd6a686da0e\") "
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127839    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-workload-socket" pod "75118b09-18bb-4678-b966-5df63a1720bf" (UID: "75118b09-18bb-4678-b966-5df63a1720bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127891    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-workload-certs\") pod \"c030a78c-6e98-43a0-927b-a4caee58d9fa\" (UID: \"c030a78c-6e98-43a0-927b-a4caee58d9fa\") "
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127930    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-credential-socket\") pod \"e764760a-aad7-44be-9068-387287c0602c\" (UID: \"e764760a-aad7-44be-9068-387287c0602c\") "
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.127991    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75118b09-18bb-4678-b966-5df63a1720bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75118b09-18bb-4678-b966-5df63a1720bf-istiod-ca-cert\") pod \"75118b09-18bb-4678-b966-5df63a1720bf\" (UID: \"75118b09-18bb-4678-b966-5df63a1720bf\") "
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.128017    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-credential-socket" pod "c030a78c-6e98-43a0-927b-a4caee58d9fa" (UID: "c030a78c-6e98-43a0-927b-a4caee58d9fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.128274    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-token" pod "03d0a6b5-42e2-4180-84f2-9bd6a686da0e" (UID: "03d0a6b5-42e2-4180-84f2-9bd6a686da0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.128690    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-workload-certs" pod "e764760a-aad7-44be-9068-387287c0602c" (UID: "e764760a-aad7-44be-9068-387287c0602c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.128921    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-data" pod "03d0a6b5-42e2-4180-84f2-9bd6a686da0e" (UID: "03d0a6b5-42e2-4180-84f2-9bd6a686da0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.129174    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-workload-socket" pod "e764760a-aad7-44be-9068-387287c0602c" (UID: "e764760a-aad7-44be-9068-387287c0602c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.129475    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75118b09-18bb-4678-b966-5df63a1720bf-istiod-ca-cert" pod "75118b09-18bb-4678-b966-5df63a1720bf" (UID: "75118b09-18bb-4678-b966-5df63a1720bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.129695    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-istio-data" pod "e764760a-aad7-44be-9068-387287c0602c" (UID: "e764760a-aad7-44be-9068-387287c0602c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.129893    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-data" pod "1638dd27-e6e3-49e0-83cd-95e91893aa1c" (UID: "1638dd27-e6e3-49e0-83cd-95e91893aa1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130361    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130385    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c030a78c-6e98-43a0-927b-a4caee58d9fa-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130400    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130412    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130425    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130438    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130453    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130467    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130481    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130494    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6vg6\" (UniqueName: \"kubernetes.io/projected/55372fd8-ea01-4397-be34-ab2260d9ef33-kube-api-access-t6vg6\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130507    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75118b09-18bb-4678-b966-5df63a1720bf-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130520    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130533    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130546    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130559    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130573    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130588    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130601    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130614    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130626    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130640    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130652    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130666    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130680    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130693    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130723    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.130841 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.130737    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55372fd8-ea01-4397-be34-ab2260d9ef33-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.133997 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.133128    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istiod-ca-cert" pod "03d0a6b5-42e2-4180-84f2-9bd6a686da0e" (UID: "03d0a6b5-42e2-4180-84f2-9bd6a686da0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:33.133997 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.133285    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-data" pod "99e50152-55fb-4e62-a4af-0efbb35fc8df" (UID: "99e50152-55fb-4e62-a4af-0efbb35fc8df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.133997 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.133365    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-envoy" pod "cd226360-69bc-45d8-95f6-9e2d00e9e4ce" (UID: "cd226360-69bc-45d8-95f6-9e2d00e9e4ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.133997 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.133400    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-workload-socket" pod "c030a78c-6e98-43a0-927b-a4caee58d9fa" (UID: "c030a78c-6e98-43a0-927b-a4caee58d9fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.133997 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.133409    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e764760a-aad7-44be-9068-387287c0602c-istio-podinfo" pod "e764760a-aad7-44be-9068-387287c0602c" (UID: "e764760a-aad7-44be-9068-387287c0602c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:33.133997 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.133484    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-token" pod "99e50152-55fb-4e62-a4af-0efbb35fc8df" (UID: "99e50152-55fb-4e62-a4af-0efbb35fc8df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:33.133997 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.133531    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-podinfo" pod "03d0a6b5-42e2-4180-84f2-9bd6a686da0e" (UID: "03d0a6b5-42e2-4180-84f2-9bd6a686da0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:33.133997 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.133638    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-workload-certs" pod "cd226360-69bc-45d8-95f6-9e2d00e9e4ce" (UID: "cd226360-69bc-45d8-95f6-9e2d00e9e4ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.133997 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.133970    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-istio-envoy" pod "75118b09-18bb-4678-b966-5df63a1720bf" (UID: "75118b09-18bb-4678-b966-5df63a1720bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.135184 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.134427    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e764760a-aad7-44be-9068-387287c0602c-istiod-ca-cert" pod "e764760a-aad7-44be-9068-387287c0602c" (UID: "e764760a-aad7-44be-9068-387287c0602c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:33.135184 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.134931    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-podinfo" pod "c030a78c-6e98-43a0-927b-a4caee58d9fa" (UID: "c030a78c-6e98-43a0-927b-a4caee58d9fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:33.135329 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.135182    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-credential-socket" pod "03d0a6b5-42e2-4180-84f2-9bd6a686da0e" (UID: "03d0a6b5-42e2-4180-84f2-9bd6a686da0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.135936 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.135458    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-workload-certs" pod "c030a78c-6e98-43a0-927b-a4caee58d9fa" (UID: "c030a78c-6e98-43a0-927b-a4caee58d9fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.135936 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.135778    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istiod-ca-cert" pod "cd226360-69bc-45d8-95f6-9e2d00e9e4ce" (UID: "cd226360-69bc-45d8-95f6-9e2d00e9e4ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:33.137304 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.137279    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-workload-certs" pod "99e50152-55fb-4e62-a4af-0efbb35fc8df" (UID: "99e50152-55fb-4e62-a4af-0efbb35fc8df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.139495 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.139313    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e50152-55fb-4e62-a4af-0efbb35fc8df-istiod-ca-cert" pod "99e50152-55fb-4e62-a4af-0efbb35fc8df" (UID: "99e50152-55fb-4e62-a4af-0efbb35fc8df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:33.139495 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.139486    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-podinfo" pod "1638dd27-e6e3-49e0-83cd-95e91893aa1c" (UID: "1638dd27-e6e3-49e0-83cd-95e91893aa1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:33.139656 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.139575    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-kube-api-access-w6wrw" pod "03d0a6b5-42e2-4180-84f2-9bd6a686da0e" (UID: "03d0a6b5-42e2-4180-84f2-9bd6a686da0e"). InnerVolumeSpecName "kube-api-access-w6wrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:33.139656 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.139594    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-credential-socket" pod "e764760a-aad7-44be-9068-387287c0602c" (UID: "e764760a-aad7-44be-9068-387287c0602c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.141535 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.141509    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-podinfo" pod "99e50152-55fb-4e62-a4af-0efbb35fc8df" (UID: "99e50152-55fb-4e62-a4af-0efbb35fc8df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:33.141628 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.141585    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75118b09-18bb-4678-b966-5df63a1720bf-kube-api-access-48cbl" pod "75118b09-18bb-4678-b966-5df63a1720bf" (UID: "75118b09-18bb-4678-b966-5df63a1720bf"). InnerVolumeSpecName "kube-api-access-48cbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:33.141747 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.141627    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-token" pod "cd226360-69bc-45d8-95f6-9e2d00e9e4ce" (UID: "cd226360-69bc-45d8-95f6-9e2d00e9e4ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:33.142091 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.142070    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-kube-api-access-n67wh" pod "cd226360-69bc-45d8-95f6-9e2d00e9e4ce" (UID: "cd226360-69bc-45d8-95f6-9e2d00e9e4ce"). InnerVolumeSpecName "kube-api-access-n67wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:33.143372 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.143268    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e764760a-aad7-44be-9068-387287c0602c-kube-api-access-rxbh6" pod "e764760a-aad7-44be-9068-387287c0602c" (UID: "e764760a-aad7-44be-9068-387287c0602c"). InnerVolumeSpecName "kube-api-access-rxbh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:33.143372 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.143297    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e764760a-aad7-44be-9068-387287c0602c-istio-token" pod "e764760a-aad7-44be-9068-387287c0602c" (UID: "e764760a-aad7-44be-9068-387287c0602c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:33.143372 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.143309    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-envoy" pod "99e50152-55fb-4e62-a4af-0efbb35fc8df" (UID: "99e50152-55fb-4e62-a4af-0efbb35fc8df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.143372 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.143331    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1638dd27-e6e3-49e0-83cd-95e91893aa1c-kube-api-access-wdz7b" pod "1638dd27-e6e3-49e0-83cd-95e91893aa1c" (UID: "1638dd27-e6e3-49e0-83cd-95e91893aa1c"). InnerVolumeSpecName "kube-api-access-wdz7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:33.143372 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.143335    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-podinfo" pod "cd226360-69bc-45d8-95f6-9e2d00e9e4ce" (UID: "cd226360-69bc-45d8-95f6-9e2d00e9e4ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:33.143372 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.143345    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75118b09-18bb-4678-b966-5df63a1720bf-istio-podinfo" pod "75118b09-18bb-4678-b966-5df63a1720bf" (UID: "75118b09-18bb-4678-b966-5df63a1720bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:33.143733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.143466    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c030a78c-6e98-43a0-927b-a4caee58d9fa-kube-api-access-czm6t" pod "c030a78c-6e98-43a0-927b-a4caee58d9fa" (UID: "c030a78c-6e98-43a0-927b-a4caee58d9fa"). InnerVolumeSpecName "kube-api-access-czm6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:33.143733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.143481    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-envoy" pod "c030a78c-6e98-43a0-927b-a4caee58d9fa" (UID: "c030a78c-6e98-43a0-927b-a4caee58d9fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.143733 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.143565    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-istio-envoy" pod "e764760a-aad7-44be-9068-387287c0602c" (UID: "e764760a-aad7-44be-9068-387287c0602c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.143922 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.143901    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-token" pod "1638dd27-e6e3-49e0-83cd-95e91893aa1c" (UID: "1638dd27-e6e3-49e0-83cd-95e91893aa1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:33.144056 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.144035    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-envoy" pod "03d0a6b5-42e2-4180-84f2-9bd6a686da0e" (UID: "03d0a6b5-42e2-4180-84f2-9bd6a686da0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.147193 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.147165    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e50152-55fb-4e62-a4af-0efbb35fc8df-kube-api-access-nt4f9" pod "99e50152-55fb-4e62-a4af-0efbb35fc8df" (UID: "99e50152-55fb-4e62-a4af-0efbb35fc8df"). InnerVolumeSpecName "kube-api-access-nt4f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:33.147275 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.147197    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-token" pod "c030a78c-6e98-43a0-927b-a4caee58d9fa" (UID: "c030a78c-6e98-43a0-927b-a4caee58d9fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:33.147275 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.147214    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-envoy" pod "1638dd27-e6e3-49e0-83cd-95e91893aa1c" (UID: "1638dd27-e6e3-49e0-83cd-95e91893aa1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:33.147362 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.147349    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75118b09-18bb-4678-b966-5df63a1720bf-istio-token" pod "75118b09-18bb-4678-b966-5df63a1720bf" (UID: "75118b09-18bb-4678-b966-5df63a1720bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:33.235588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.233480    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.235588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.233513    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.235588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.233525    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75118b09-18bb-4678-b966-5df63a1720bf-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.235588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.233538    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.235588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.233551    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75118b09-18bb-4678-b966-5df63a1720bf-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.235588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.233563    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxbh6\" (UniqueName: \"kubernetes.io/projected/e764760a-aad7-44be-9068-387287c0602c-kube-api-access-rxbh6\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.235588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.233579    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.235588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.233592    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.235588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.233603    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.235588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.233615    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.235588 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.233627    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236193 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236171    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236269 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236197    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48cbl\" (UniqueName: \"kubernetes.io/projected/75118b09-18bb-4678-b966-5df63a1720bf-kube-api-access-48cbl\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236269 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236212    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236269 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236229    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236269 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236242    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e764760a-aad7-44be-9068-387287c0602c-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236269 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236254    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236269 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236267    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e50152-55fb-4e62-a4af-0efbb35fc8df-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236280    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236293    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czm6t\" (UniqueName: \"kubernetes.io/projected/c030a78c-6e98-43a0-927b-a4caee58d9fa-kube-api-access-czm6t\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236309    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75118b09-18bb-4678-b966-5df63a1720bf-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236323    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236339    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6wrw\" (UniqueName: \"kubernetes.io/projected/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-kube-api-access-w6wrw\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236353    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e50152-55fb-4e62-a4af-0efbb35fc8df-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236367    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n67wh\" (UniqueName: \"kubernetes.io/projected/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-kube-api-access-n67wh\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236380    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt4f9\" (UniqueName: \"kubernetes.io/projected/99e50152-55fb-4e62-a4af-0efbb35fc8df-kube-api-access-nt4f9\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236394    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236407    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e764760a-aad7-44be-9068-387287c0602c-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236422    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236435    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1638dd27-e6e3-49e0-83cd-95e91893aa1c-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236449    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdz7b\" (UniqueName: \"kubernetes.io/projected/1638dd27-e6e3-49e0-83cd-95e91893aa1c-kube-api-access-wdz7b\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236463    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0a6b5-42e2-4180-84f2-9bd6a686da0e-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236477    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c030a78c-6e98-43a0-927b-a4caee58d9fa-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236490    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764760a-aad7-44be-9068-387287c0602c-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236505    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e764760a-aad7-44be-9068-387287c0602c-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236519    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c030a78c-6e98-43a0-927b-a4caee58d9fa-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236531    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd226360-69bc-45d8-95f6-9e2d00e9e4ce-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.236574 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.236544    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e50152-55fb-4e62-a4af-0efbb35fc8df-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:33.274943 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.274903    6099 generic.go:356] "Generic (PLEG): container finished" podID="03d0a6b5-42e2-4180-84f2-9bd6a686da0e" containerID="d3b278a654e7d3f80a3faf005e2a447d9b2bbe969694b0c33579f7bfc931a7b0" exitCode=0
Oct 30 00:59:33.275106 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.274955    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd" event={"ID":"03d0a6b5-42e2-4180-84f2-9bd6a686da0e","Type":"ContainerDied","Data":"d3b278a654e7d3f80a3faf005e2a447d9b2bbe969694b0c33579f7bfc931a7b0"}
Oct 30 00:59:33.275106 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.274965    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"
Oct 30 00:59:33.275106 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.275028    6099 scope.go:122] "RemoveContainer" containerID="d3b278a654e7d3f80a3faf005e2a447d9b2bbe969694b0c33579f7bfc931a7b0"
Oct 30 00:59:33.275106 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.275014    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd" event={"ID":"03d0a6b5-42e2-4180-84f2-9bd6a686da0e","Type":"ContainerDied","Data":"25e04858b66f1ad7e6a7ce57f87ac491a1e14089b647aadc928f4825f14c58fd"}
Oct 30 00:59:33.276838 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.276731    6099 generic.go:356] "Generic (PLEG): container finished" podID="75118b09-18bb-4678-b966-5df63a1720bf" containerID="8d06aa87ba2dd99a82f43b6f6f8f45a732298415973e5ec73cc89e7193721860" exitCode=0
Oct 30 00:59:33.276838 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.276805    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm" event={"ID":"75118b09-18bb-4678-b966-5df63a1720bf","Type":"ContainerDied","Data":"8d06aa87ba2dd99a82f43b6f6f8f45a732298415973e5ec73cc89e7193721860"}
Oct 30 00:59:33.276838 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.276817    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"
Oct 30 00:59:33.277042 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.276846    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm" event={"ID":"75118b09-18bb-4678-b966-5df63a1720bf","Type":"ContainerDied","Data":"2c061ea7e15181520cc39ae0f330be3830386252b4131c992cdff7977f1d1897"}
Oct 30 00:59:33.278474 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.278449    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc" event={"ID":"55372fd8-ea01-4397-be34-ab2260d9ef33","Type":"ContainerDied","Data":"b71a87a0aa36ee7ae8887740d836e2d0968a343ac24effc4f01ced0847341bd2"}
Oct 30 00:59:33.278595 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.278499    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b71a87a0aa36ee7ae8887740d836e2d0968a343ac24effc4f01ced0847341bd2"
Oct 30 00:59:33.278595 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.278551    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc"
Oct 30 00:59:33.286829 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.286636    6099 generic.go:356] "Generic (PLEG): container finished" podID="99e50152-55fb-4e62-a4af-0efbb35fc8df" containerID="91a62bc9e59c814f5393173a63c87772d25d24af64291d5ff43c81d486a29b0f" exitCode=0
Oct 30 00:59:33.286829 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.286682    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"
Oct 30 00:59:33.286829 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.286734    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv" event={"ID":"99e50152-55fb-4e62-a4af-0efbb35fc8df","Type":"ContainerDied","Data":"91a62bc9e59c814f5393173a63c87772d25d24af64291d5ff43c81d486a29b0f"}
Oct 30 00:59:33.286829 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.286785    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv" event={"ID":"99e50152-55fb-4e62-a4af-0efbb35fc8df","Type":"ContainerDied","Data":"94e5a4162750393ba1fb1e29ab4364bfe32837e2ca6f8b19019df6961fd2e032"}
Oct 30 00:59:33.288923 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.288900    6099 generic.go:356] "Generic (PLEG): container finished" podID="cd226360-69bc-45d8-95f6-9e2d00e9e4ce" containerID="3568018eeae415a06f67b2380ae899c9d32dbedc4a95fc1dd9272b44dc12cbe8" exitCode=0
Oct 30 00:59:33.289031 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.288946    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"
Oct 30 00:59:33.289357 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.289128    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp" event={"ID":"cd226360-69bc-45d8-95f6-9e2d00e9e4ce","Type":"ContainerDied","Data":"3568018eeae415a06f67b2380ae899c9d32dbedc4a95fc1dd9272b44dc12cbe8"}
Oct 30 00:59:33.289357 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.289179    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp" event={"ID":"cd226360-69bc-45d8-95f6-9e2d00e9e4ce","Type":"ContainerDied","Data":"db5b381a165cc918caf2862ec3966fa091882828b05a7af5201f2e009e95913a"}
Oct 30 00:59:33.300669 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.300636    6099 generic.go:356] "Generic (PLEG): container finished" podID="c030a78c-6e98-43a0-927b-a4caee58d9fa" containerID="344c718722a9b7d9c4ba28ef84052530801ecec077961775300540c5df0d73e1" exitCode=0
Oct 30 00:59:33.300922 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.300899    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5" event={"ID":"c030a78c-6e98-43a0-927b-a4caee58d9fa","Type":"ContainerDied","Data":"344c718722a9b7d9c4ba28ef84052530801ecec077961775300540c5df0d73e1"}
Oct 30 00:59:33.301021 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.300955    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5" event={"ID":"c030a78c-6e98-43a0-927b-a4caee58d9fa","Type":"ContainerDied","Data":"e07bd4d9bc07ab136ec083fb3cf03eb3232b2cfe9ce4c2544bab0318cb5d3916"}
Oct 30 00:59:33.301098 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.301036    6099 scope.go:122] "RemoveContainer" containerID="d3b278a654e7d3f80a3faf005e2a447d9b2bbe969694b0c33579f7bfc931a7b0"
Oct 30 00:59:33.301153 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.301133    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"
Oct 30 00:59:33.305962 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:59:33.305935    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b278a654e7d3f80a3faf005e2a447d9b2bbe969694b0c33579f7bfc931a7b0\": container with ID starting with d3b278a654e7d3f80a3faf005e2a447d9b2bbe969694b0c33579f7bfc931a7b0 not found: ID does not exist" containerID="d3b278a654e7d3f80a3faf005e2a447d9b2bbe969694b0c33579f7bfc931a7b0"
Oct 30 00:59:33.306124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.306091    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b278a654e7d3f80a3faf005e2a447d9b2bbe969694b0c33579f7bfc931a7b0"} err="failed to get container status \"d3b278a654e7d3f80a3faf005e2a447d9b2bbe969694b0c33579f7bfc931a7b0\": rpc error: code = NotFound desc = could not find container \"d3b278a654e7d3f80a3faf005e2a447d9b2bbe969694b0c33579f7bfc931a7b0\": container with ID starting with d3b278a654e7d3f80a3faf005e2a447d9b2bbe969694b0c33579f7bfc931a7b0 not found: ID does not exist"
Oct 30 00:59:33.306124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.306125    6099 scope.go:122] "RemoveContainer" containerID="8d06aa87ba2dd99a82f43b6f6f8f45a732298415973e5ec73cc89e7193721860"
Oct 30 00:59:33.308328 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.308301    6099 generic.go:356] "Generic (PLEG): container finished" podID="e764760a-aad7-44be-9068-387287c0602c" containerID="d9fac4f2d3efb6e291959cba6511e86001b2117fba7e6efc6f7452822f6c4dcd" exitCode=0
Oct 30 00:59:33.308421 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.308371    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx" event={"ID":"e764760a-aad7-44be-9068-387287c0602c","Type":"ContainerDied","Data":"d9fac4f2d3efb6e291959cba6511e86001b2117fba7e6efc6f7452822f6c4dcd"}
Oct 30 00:59:33.308421 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.308405    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx" event={"ID":"e764760a-aad7-44be-9068-387287c0602c","Type":"ContainerDied","Data":"b42959161a4b55da4bbeaa7913e4ce536211a5962cccd152877b2362c01cadfe"}
Oct 30 00:59:33.308552 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.308461    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"
Oct 30 00:59:33.311379 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.311328    6099 generic.go:356] "Generic (PLEG): container finished" podID="1638dd27-e6e3-49e0-83cd-95e91893aa1c" containerID="2e496e88c9af041b98e19ab627d12407f0d703d70850ab109fdf8c69a13b63d8" exitCode=0
Oct 30 00:59:33.311379 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.311363    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj" event={"ID":"1638dd27-e6e3-49e0-83cd-95e91893aa1c","Type":"ContainerDied","Data":"2e496e88c9af041b98e19ab627d12407f0d703d70850ab109fdf8c69a13b63d8"}
Oct 30 00:59:33.311495 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.311407    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj" event={"ID":"1638dd27-e6e3-49e0-83cd-95e91893aa1c","Type":"ContainerDied","Data":"da365b0300b77635c1db62b079e45eb4b8e2bb5b2f965e7a2f63658d45ee0cf4"}
Oct 30 00:59:33.311495 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.311471    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"
Oct 30 00:59:33.321566 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.321458    6099 scope.go:122] "RemoveContainer" containerID="8d06aa87ba2dd99a82f43b6f6f8f45a732298415973e5ec73cc89e7193721860"
Oct 30 00:59:33.321854 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:59:33.321833    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d06aa87ba2dd99a82f43b6f6f8f45a732298415973e5ec73cc89e7193721860\": container with ID starting with 8d06aa87ba2dd99a82f43b6f6f8f45a732298415973e5ec73cc89e7193721860 not found: ID does not exist" containerID="8d06aa87ba2dd99a82f43b6f6f8f45a732298415973e5ec73cc89e7193721860"
Oct 30 00:59:33.321941 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.321863    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d06aa87ba2dd99a82f43b6f6f8f45a732298415973e5ec73cc89e7193721860"} err="failed to get container status \"8d06aa87ba2dd99a82f43b6f6f8f45a732298415973e5ec73cc89e7193721860\": rpc error: code = NotFound desc = could not find container \"8d06aa87ba2dd99a82f43b6f6f8f45a732298415973e5ec73cc89e7193721860\": container with ID starting with 8d06aa87ba2dd99a82f43b6f6f8f45a732298415973e5ec73cc89e7193721860 not found: ID does not exist"
Oct 30 00:59:33.321941 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.321891    6099 scope.go:122] "RemoveContainer" containerID="91a62bc9e59c814f5393173a63c87772d25d24af64291d5ff43c81d486a29b0f"
Oct 30 00:59:33.335277 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.335257    6099 scope.go:122] "RemoveContainer" containerID="91a62bc9e59c814f5393173a63c87772d25d24af64291d5ff43c81d486a29b0f"
Oct 30 00:59:33.335558 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:59:33.335540    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a62bc9e59c814f5393173a63c87772d25d24af64291d5ff43c81d486a29b0f\": container with ID starting with 91a62bc9e59c814f5393173a63c87772d25d24af64291d5ff43c81d486a29b0f not found: ID does not exist" containerID="91a62bc9e59c814f5393173a63c87772d25d24af64291d5ff43c81d486a29b0f"
Oct 30 00:59:33.335621 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.335566    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a62bc9e59c814f5393173a63c87772d25d24af64291d5ff43c81d486a29b0f"} err="failed to get container status \"91a62bc9e59c814f5393173a63c87772d25d24af64291d5ff43c81d486a29b0f\": rpc error: code = NotFound desc = could not find container \"91a62bc9e59c814f5393173a63c87772d25d24af64291d5ff43c81d486a29b0f\": container with ID starting with 91a62bc9e59c814f5393173a63c87772d25d24af64291d5ff43c81d486a29b0f not found: ID does not exist"
Oct 30 00:59:33.335621 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.335583    6099 scope.go:122] "RemoveContainer" containerID="3568018eeae415a06f67b2380ae899c9d32dbedc4a95fc1dd9272b44dc12cbe8"
Oct 30 00:59:33.347678 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.347661    6099 scope.go:122] "RemoveContainer" containerID="3568018eeae415a06f67b2380ae899c9d32dbedc4a95fc1dd9272b44dc12cbe8"
Oct 30 00:59:33.348270 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:59:33.348099    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3568018eeae415a06f67b2380ae899c9d32dbedc4a95fc1dd9272b44dc12cbe8\": container with ID starting with 3568018eeae415a06f67b2380ae899c9d32dbedc4a95fc1dd9272b44dc12cbe8 not found: ID does not exist" containerID="3568018eeae415a06f67b2380ae899c9d32dbedc4a95fc1dd9272b44dc12cbe8"
Oct 30 00:59:33.348270 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.348151    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3568018eeae415a06f67b2380ae899c9d32dbedc4a95fc1dd9272b44dc12cbe8"} err="failed to get container status \"3568018eeae415a06f67b2380ae899c9d32dbedc4a95fc1dd9272b44dc12cbe8\": rpc error: code = NotFound desc = could not find container \"3568018eeae415a06f67b2380ae899c9d32dbedc4a95fc1dd9272b44dc12cbe8\": container with ID starting with 3568018eeae415a06f67b2380ae899c9d32dbedc4a95fc1dd9272b44dc12cbe8 not found: ID does not exist"
Oct 30 00:59:33.348270 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.348173    6099 scope.go:122] "RemoveContainer" containerID="344c718722a9b7d9c4ba28ef84052530801ecec077961775300540c5df0d73e1"
Oct 30 00:59:33.361196 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.361174    6099 scope.go:122] "RemoveContainer" containerID="344c718722a9b7d9c4ba28ef84052530801ecec077961775300540c5df0d73e1"
Oct 30 00:59:33.361497 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:59:33.361476    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344c718722a9b7d9c4ba28ef84052530801ecec077961775300540c5df0d73e1\": container with ID starting with 344c718722a9b7d9c4ba28ef84052530801ecec077961775300540c5df0d73e1 not found: ID does not exist" containerID="344c718722a9b7d9c4ba28ef84052530801ecec077961775300540c5df0d73e1"
Oct 30 00:59:33.361584 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.361511    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344c718722a9b7d9c4ba28ef84052530801ecec077961775300540c5df0d73e1"} err="failed to get container status \"344c718722a9b7d9c4ba28ef84052530801ecec077961775300540c5df0d73e1\": rpc error: code = NotFound desc = could not find container \"344c718722a9b7d9c4ba28ef84052530801ecec077961775300540c5df0d73e1\": container with ID starting with 344c718722a9b7d9c4ba28ef84052530801ecec077961775300540c5df0d73e1 not found: ID does not exist"
Oct 30 00:59:33.361584 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.361534    6099 scope.go:122] "RemoveContainer" containerID="d9fac4f2d3efb6e291959cba6511e86001b2117fba7e6efc6f7452822f6c4dcd"
Oct 30 00:59:33.372892 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.372865    6099 scope.go:122] "RemoveContainer" containerID="4a3343d4a0da3b853c0b48dc5bb854a0fe56ba874dc6655339096fd7f6b7050a"
Oct 30 00:59:33.384205 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.384118    6099 scope.go:122] "RemoveContainer" containerID="d9fac4f2d3efb6e291959cba6511e86001b2117fba7e6efc6f7452822f6c4dcd"
Oct 30 00:59:33.384445 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:59:33.384420    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9fac4f2d3efb6e291959cba6511e86001b2117fba7e6efc6f7452822f6c4dcd\": container with ID starting with d9fac4f2d3efb6e291959cba6511e86001b2117fba7e6efc6f7452822f6c4dcd not found: ID does not exist" containerID="d9fac4f2d3efb6e291959cba6511e86001b2117fba7e6efc6f7452822f6c4dcd"
Oct 30 00:59:33.384507 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.384455    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9fac4f2d3efb6e291959cba6511e86001b2117fba7e6efc6f7452822f6c4dcd"} err="failed to get container status \"d9fac4f2d3efb6e291959cba6511e86001b2117fba7e6efc6f7452822f6c4dcd\": rpc error: code = NotFound desc = could not find container \"d9fac4f2d3efb6e291959cba6511e86001b2117fba7e6efc6f7452822f6c4dcd\": container with ID starting with d9fac4f2d3efb6e291959cba6511e86001b2117fba7e6efc6f7452822f6c4dcd not found: ID does not exist"
Oct 30 00:59:33.384507 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.384474    6099 scope.go:122] "RemoveContainer" containerID="4a3343d4a0da3b853c0b48dc5bb854a0fe56ba874dc6655339096fd7f6b7050a"
Oct 30 00:59:33.384800 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:59:33.384779    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3343d4a0da3b853c0b48dc5bb854a0fe56ba874dc6655339096fd7f6b7050a\": container with ID starting with 4a3343d4a0da3b853c0b48dc5bb854a0fe56ba874dc6655339096fd7f6b7050a not found: ID does not exist" containerID="4a3343d4a0da3b853c0b48dc5bb854a0fe56ba874dc6655339096fd7f6b7050a"
Oct 30 00:59:33.385209 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.384809    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3343d4a0da3b853c0b48dc5bb854a0fe56ba874dc6655339096fd7f6b7050a"} err="failed to get container status \"4a3343d4a0da3b853c0b48dc5bb854a0fe56ba874dc6655339096fd7f6b7050a\": rpc error: code = NotFound desc = could not find container \"4a3343d4a0da3b853c0b48dc5bb854a0fe56ba874dc6655339096fd7f6b7050a\": container with ID starting with 4a3343d4a0da3b853c0b48dc5bb854a0fe56ba874dc6655339096fd7f6b7050a not found: ID does not exist"
Oct 30 00:59:33.385209 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.384824    6099 scope.go:122] "RemoveContainer" containerID="2e496e88c9af041b98e19ab627d12407f0d703d70850ab109fdf8c69a13b63d8"
Oct 30 00:59:33.431867 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.431834    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"]
Oct 30 00:59:33.439683 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.439653    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4hkcx"]
Oct 30 00:59:33.475498 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.475464    6099 scope.go:122] "RemoveContainer" containerID="2e496e88c9af041b98e19ab627d12407f0d703d70850ab109fdf8c69a13b63d8"
Oct 30 00:59:33.478855 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:59:33.477034    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e496e88c9af041b98e19ab627d12407f0d703d70850ab109fdf8c69a13b63d8\": container with ID starting with 2e496e88c9af041b98e19ab627d12407f0d703d70850ab109fdf8c69a13b63d8 not found: ID does not exist" containerID="2e496e88c9af041b98e19ab627d12407f0d703d70850ab109fdf8c69a13b63d8"
Oct 30 00:59:33.478994 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.478864    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e496e88c9af041b98e19ab627d12407f0d703d70850ab109fdf8c69a13b63d8"} err="failed to get container status \"2e496e88c9af041b98e19ab627d12407f0d703d70850ab109fdf8c69a13b63d8\": rpc error: code = NotFound desc = could not find container \"2e496e88c9af041b98e19ab627d12407f0d703d70850ab109fdf8c69a13b63d8\": container with ID starting with 2e496e88c9af041b98e19ab627d12407f0d703d70850ab109fdf8c69a13b63d8 not found: ID does not exist"
Oct 30 00:59:33.490802 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.490770    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"]
Oct 30 00:59:33.493185 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.493161    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-49jlj"]
Oct 30 00:59:33.520039 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.520012    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"]
Oct 30 00:59:33.523738 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.523685    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-hn5lv"]
Oct 30 00:59:33.571352 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.571318    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"]
Oct 30 00:59:33.576958 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.576934    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-2cwp5"]
Oct 30 00:59:33.618061 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.618031    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"]
Oct 30 00:59:33.627953 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.627894    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-qqsbd"]
Oct 30 00:59:33.663548 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.663513    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"]
Oct 30 00:59:33.671605 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.671575    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-95dtm"]
Oct 30 00:59:33.704695 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.704667    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"]
Oct 30 00:59:33.711038 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:33.711011    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a4fe0143-openshift-default-69d54b6bdf-4gkhp"]
Oct 30 00:59:34.263589 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.263556    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03d0a6b5-42e2-4180-84f2-9bd6a686da0e" path="/var/lib/kubelet/pods/03d0a6b5-42e2-4180-84f2-9bd6a686da0e/volumes"
Oct 30 00:59:34.264149 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.264132    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1638dd27-e6e3-49e0-83cd-95e91893aa1c" path="/var/lib/kubelet/pods/1638dd27-e6e3-49e0-83cd-95e91893aa1c/volumes"
Oct 30 00:59:34.264627 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.264613    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75118b09-18bb-4678-b966-5df63a1720bf" path="/var/lib/kubelet/pods/75118b09-18bb-4678-b966-5df63a1720bf/volumes"
Oct 30 00:59:34.265440 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.265424    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99e50152-55fb-4e62-a4af-0efbb35fc8df" path="/var/lib/kubelet/pods/99e50152-55fb-4e62-a4af-0efbb35fc8df/volumes"
Oct 30 00:59:34.265993 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.265972    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c030a78c-6e98-43a0-927b-a4caee58d9fa" path="/var/lib/kubelet/pods/c030a78c-6e98-43a0-927b-a4caee58d9fa/volumes"
Oct 30 00:59:34.266626 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.266603    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd226360-69bc-45d8-95f6-9e2d00e9e4ce" path="/var/lib/kubelet/pods/cd226360-69bc-45d8-95f6-9e2d00e9e4ce/volumes"
Oct 30 00:59:34.267679 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.267660    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e764760a-aad7-44be-9068-387287c0602c" path="/var/lib/kubelet/pods/e764760a-aad7-44be-9068-387287c0602c/volumes"
Oct 30 00:59:34.629971 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.629939    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h"]
Oct 30 00:59:34.630640 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630626    6099 cpu_manager.go:414] "containerMap: removing container" podUID="99e50152-55fb-4e62-a4af-0efbb35fc8df" containerName="istio-proxy"
Oct 30 00:59:34.630687 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630642    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99e50152-55fb-4e62-a4af-0efbb35fc8df" containerName="istio-proxy"
Oct 30 00:59:34.630687 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630656    6099 cpu_manager.go:414] "containerMap: removing container" podUID="1638dd27-e6e3-49e0-83cd-95e91893aa1c" containerName="istio-proxy"
Oct 30 00:59:34.630687 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630661    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1638dd27-e6e3-49e0-83cd-95e91893aa1c" containerName="istio-proxy"
Oct 30 00:59:34.630687 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630678    6099 cpu_manager.go:414] "containerMap: removing container" podUID="03d0a6b5-42e2-4180-84f2-9bd6a686da0e" containerName="istio-proxy"
Oct 30 00:59:34.630687 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630683    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03d0a6b5-42e2-4180-84f2-9bd6a686da0e" containerName="istio-proxy"
Oct 30 00:59:34.630873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630717    6099 cpu_manager.go:414] "containerMap: removing container" podUID="cd226360-69bc-45d8-95f6-9e2d00e9e4ce" containerName="istio-proxy"
Oct 30 00:59:34.630873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630725    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd226360-69bc-45d8-95f6-9e2d00e9e4ce" containerName="istio-proxy"
Oct 30 00:59:34.630873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630751    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e764760a-aad7-44be-9068-387287c0602c" containerName="istio-proxy"
Oct 30 00:59:34.630873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630757    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e764760a-aad7-44be-9068-387287c0602c" containerName="istio-proxy"
Oct 30 00:59:34.630873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630763    6099 cpu_manager.go:414] "containerMap: removing container" podUID="c030a78c-6e98-43a0-927b-a4caee58d9fa" containerName="istio-proxy"
Oct 30 00:59:34.630873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630767    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c030a78c-6e98-43a0-927b-a4caee58d9fa" containerName="istio-proxy"
Oct 30 00:59:34.630873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630782    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e764760a-aad7-44be-9068-387287c0602c" containerName="istio-proxy"
Oct 30 00:59:34.630873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630786    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e764760a-aad7-44be-9068-387287c0602c" containerName="istio-proxy"
Oct 30 00:59:34.630873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630796    6099 cpu_manager.go:414] "containerMap: removing container" podUID="75118b09-18bb-4678-b966-5df63a1720bf" containerName="istio-proxy"
Oct 30 00:59:34.630873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630800    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75118b09-18bb-4678-b966-5df63a1720bf" containerName="istio-proxy"
Oct 30 00:59:34.630873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630817    6099 cpu_manager.go:414] "containerMap: removing container" podUID="55372fd8-ea01-4397-be34-ab2260d9ef33" containerName="revoke-keys"
Oct 30 00:59:34.630873 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630824    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55372fd8-ea01-4397-be34-ab2260d9ef33" containerName="revoke-keys"
Oct 30 00:59:34.631244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630944    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd226360-69bc-45d8-95f6-9e2d00e9e4ce" containerName="istio-proxy"
Oct 30 00:59:34.631244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630956    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e764760a-aad7-44be-9068-387287c0602c" containerName="istio-proxy"
Oct 30 00:59:34.631244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630961    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c030a78c-6e98-43a0-927b-a4caee58d9fa" containerName="istio-proxy"
Oct 30 00:59:34.631244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630968    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e764760a-aad7-44be-9068-387287c0602c" containerName="istio-proxy"
Oct 30 00:59:34.631244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630976    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="75118b09-18bb-4678-b966-5df63a1720bf" containerName="istio-proxy"
Oct 30 00:59:34.631244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630985    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="55372fd8-ea01-4397-be34-ab2260d9ef33" containerName="revoke-keys"
Oct 30 00:59:34.631244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630992    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="99e50152-55fb-4e62-a4af-0efbb35fc8df" containerName="istio-proxy"
Oct 30 00:59:34.631244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.630998    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1638dd27-e6e3-49e0-83cd-95e91893aa1c" containerName="istio-proxy"
Oct 30 00:59:34.631244 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.631006    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="03d0a6b5-42e2-4180-84f2-9bd6a686da0e" containerName="istio-proxy"
Oct 30 00:59:34.637278 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.637255    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h"
Oct 30 00:59:34.652905 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.652877    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-43c02d-dockercfg-d7tt8\""
Oct 30 00:59:34.654735 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.654681    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h"]
Oct 30 00:59:34.661893 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.661870    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-43c02d\""
Oct 30 00:59:34.694717 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.692773    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3064087b-a922-4f17-910e-697c6960c3e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h\" (UID: \"3064087b-a922-4f17-910e-697c6960c3e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h"
Oct 30 00:59:34.694717 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.692922    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v45tw\" (UniqueName: \"kubernetes.io/projected/3064087b-a922-4f17-910e-697c6960c3e4-kube-api-access-v45tw\") pod \"payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h\" (UID: \"3064087b-a922-4f17-910e-697c6960c3e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h"
Oct 30 00:59:34.803037 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.802995    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3064087b-a922-4f17-910e-697c6960c3e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h\" (UID: \"3064087b-a922-4f17-910e-697c6960c3e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h"
Oct 30 00:59:34.803249 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.803110    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v45tw\" (UniqueName: \"kubernetes.io/projected/3064087b-a922-4f17-910e-697c6960c3e4-kube-api-access-v45tw\") pod \"payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h\" (UID: \"3064087b-a922-4f17-910e-697c6960c3e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h"
Oct 30 00:59:34.804327 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.804248    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3064087b-a922-4f17-910e-697c6960c3e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h\" (UID: \"3064087b-a922-4f17-910e-697c6960c3e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h"
Oct 30 00:59:34.845945 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.845580    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn"]
Oct 30 00:59:34.866927 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.866893    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn"]
Oct 30 00:59:34.867121 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.867019    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn"
Oct 30 00:59:34.906780 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.892178    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v45tw\" (UniqueName: \"kubernetes.io/projected/3064087b-a922-4f17-910e-697c6960c3e4-kube-api-access-v45tw\") pod \"payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h\" (UID: \"3064087b-a922-4f17-910e-697c6960c3e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h"
Oct 30 00:59:34.956175 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.956140    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h"
Oct 30 00:59:34.958356 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.957526    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x22qc\" (UniqueName: \"kubernetes.io/projected/c8c2f8c2-32f1-4acb-a67a-44b18bf39464-kube-api-access-x22qc\") pod \"payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn\" (UID: \"c8c2f8c2-32f1-4acb-a67a-44b18bf39464\") " pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn"
Oct 30 00:59:34.958796 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:34.958770    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c2f8c2-32f1-4acb-a67a-44b18bf39464-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn\" (UID: \"c8c2f8c2-32f1-4acb-a67a-44b18bf39464\") " pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn"
Oct 30 00:59:35.061936 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:35.060164    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x22qc\" (UniqueName: \"kubernetes.io/projected/c8c2f8c2-32f1-4acb-a67a-44b18bf39464-kube-api-access-x22qc\") pod \"payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn\" (UID: \"c8c2f8c2-32f1-4acb-a67a-44b18bf39464\") " pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn"
Oct 30 00:59:35.061936 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:35.060538    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c2f8c2-32f1-4acb-a67a-44b18bf39464-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn\" (UID: \"c8c2f8c2-32f1-4acb-a67a-44b18bf39464\") " pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn"
Oct 30 00:59:35.061936 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:35.061516    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c2f8c2-32f1-4acb-a67a-44b18bf39464-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn\" (UID: \"c8c2f8c2-32f1-4acb-a67a-44b18bf39464\") " pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn"
Oct 30 00:59:35.084649 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:35.084595    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x22qc\" (UniqueName: \"kubernetes.io/projected/c8c2f8c2-32f1-4acb-a67a-44b18bf39464-kube-api-access-x22qc\") pod \"payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn\" (UID: \"c8c2f8c2-32f1-4acb-a67a-44b18bf39464\") " pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn"
Oct 30 00:59:35.258701 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:35.258612    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn"
Oct 30 00:59:35.282429 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:59:35.282396    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3064087b_a922_4f17_910e_697c6960c3e4.slice/crio-0d72f6c8be72e2d56f36947e6bdcc419808f13a12f966a8e95125837d2d90abc WatchSource:0}: Error finding container 0d72f6c8be72e2d56f36947e6bdcc419808f13a12f966a8e95125837d2d90abc: Status 404 returned error can't find the container with id 0d72f6c8be72e2d56f36947e6bdcc419808f13a12f966a8e95125837d2d90abc
Oct 30 00:59:35.285063 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:35.285036    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h"]
Oct 30 00:59:35.347032 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:35.346997    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h" event={"ID":"3064087b-a922-4f17-910e-697c6960c3e4","Type":"ContainerStarted","Data":"0d72f6c8be72e2d56f36947e6bdcc419808f13a12f966a8e95125837d2d90abc"}
Oct 30 00:59:35.565879 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:59:35.565847    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c2f8c2_32f1_4acb_a67a_44b18bf39464.slice/crio-6060a73843696ace308b9343effe23a739949d1746ac12d30efec7d7f9358ba3 WatchSource:0}: Error finding container 6060a73843696ace308b9343effe23a739949d1746ac12d30efec7d7f9358ba3: Status 404 returned error can't find the container with id 6060a73843696ace308b9343effe23a739949d1746ac12d30efec7d7f9358ba3
Oct 30 00:59:35.566409 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:35.566386    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn"]
Oct 30 00:59:36.366027 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:36.365973    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn" event={"ID":"c8c2f8c2-32f1-4acb-a67a-44b18bf39464","Type":"ContainerStarted","Data":"fd2435d0df5c386a00f1c8e11f28bc713086893a10380780385228d9a5e33048"}
Oct 30 00:59:36.366507 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:36.366047    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn" event={"ID":"c8c2f8c2-32f1-4acb-a67a-44b18bf39464","Type":"ContainerStarted","Data":"6060a73843696ace308b9343effe23a739949d1746ac12d30efec7d7f9358ba3"}
Oct 30 00:59:36.366507 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:36.366235    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn"
Oct 30 00:59:36.368011 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:36.367982    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h" event={"ID":"3064087b-a922-4f17-910e-697c6960c3e4","Type":"ContainerStarted","Data":"5c8a879c5d4c5d680d6e96bb067be892b122700709aae19f0b802f186a80b55d"}
Oct 30 00:59:36.368147 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:36.368102    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h"
Oct 30 00:59:36.416614 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:36.416533    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn" podStartSLOduration=2.416513428 podStartE2EDuration="2.416513428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:36.405569246 +0000 UTC m=+1470.262477001" watchObservedRunningTime="2026-10-30 00:59:36.416513428 +0000 UTC m=+1470.273421194"
Oct 30 00:59:36.452121 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:36.452061    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h" podStartSLOduration=2.452046958 podStartE2EDuration="2.452046958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:36.448001664 +0000 UTC m=+1470.304909419" watchObservedRunningTime="2026-10-30 00:59:36.452046958 +0000 UTC m=+1470.308954709"
Oct 30 00:59:39.708538 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:39.708493    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 00:59:42.449891 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:42.449844    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:43.054940 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:43.054825    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:44.167196 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:44.165119    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w"]
Oct 30 00:59:44.171181 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:44.171151    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-610065-mxq8w"]
Oct 30 00:59:44.264027 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:44.263996    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c213fa-596e-4df6-9966-ad1f6c56a30e" path="/var/lib/kubelet/pods/f1c213fa-596e-4df6-9966-ad1f6c56a30e/volumes"
Oct 30 00:59:46.632183 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:59:46.632143    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 00:59:47.380414 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:47.380385    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h"
Oct 30 00:59:47.380614 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:47.380456    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn"
Oct 30 00:59:51.596689 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:51.596650    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"]
Oct 30 00:59:51.597300 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:51.597265    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr" podUID="a9004633-73af-405e-8cd8-b1bd569602af" containerName="istio-proxy" containerID="cri-o://8df45b1fc6e1b6b783d73152c440ddb40e495192b42cf37333904d5061a90f36" gracePeriod=30
Oct 30 00:59:51.811898 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:51.811814    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4"]
Oct 30 00:59:51.817744 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:51.817696    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-353403-z2ck4"]
Oct 30 00:59:52.265754 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.265723    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5be716f-50a7-4723-acf5-9b987bf7fab8" path="/var/lib/kubelet/pods/d5be716f-50a7-4723-acf5-9b987bf7fab8/volumes"
Oct 30 00:59:52.309670 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.309635    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq"]
Oct 30 00:59:52.319991 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.317693    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq"
Oct 30 00:59:52.361778 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.361650    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq"]
Oct 30 00:59:52.373089 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.373051    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtvkj\" (UniqueName: \"kubernetes.io/projected/c60cba8a-34d4-454c-8405-9afa8c9b4f76-kube-api-access-jtvkj\") pod \"maas-api-revoke-keys-e2e-api-b-601aea-nzcvq\" (UID: \"c60cba8a-34d4-454c-8405-9afa8c9b4f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq"
Oct 30 00:59:52.373767 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.373739    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c60cba8a-34d4-454c-8405-9afa8c9b4f76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-601aea-nzcvq\" (UID: \"c60cba8a-34d4-454c-8405-9afa8c9b4f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq"
Oct 30 00:59:52.375031 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.375010    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c60cba8a-34d4-454c-8405-9afa8c9b4f76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-601aea-nzcvq\" (UID: \"c60cba8a-34d4-454c-8405-9afa8c9b4f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq"
Oct 30 00:59:52.383033 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.382830    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:59:52.383033 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.382982    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:59:52.476661 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.476627    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c60cba8a-34d4-454c-8405-9afa8c9b4f76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-601aea-nzcvq\" (UID: \"c60cba8a-34d4-454c-8405-9afa8c9b4f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq"
Oct 30 00:59:52.476876 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.476684    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtvkj\" (UniqueName: \"kubernetes.io/projected/c60cba8a-34d4-454c-8405-9afa8c9b4f76-kube-api-access-jtvkj\") pod \"maas-api-revoke-keys-e2e-api-b-601aea-nzcvq\" (UID: \"c60cba8a-34d4-454c-8405-9afa8c9b4f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq"
Oct 30 00:59:52.476992 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.476924    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c60cba8a-34d4-454c-8405-9afa8c9b4f76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-601aea-nzcvq\" (UID: \"c60cba8a-34d4-454c-8405-9afa8c9b4f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq"
Oct 30 00:59:52.477522 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.477493    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c60cba8a-34d4-454c-8405-9afa8c9b4f76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-601aea-nzcvq\" (UID: \"c60cba8a-34d4-454c-8405-9afa8c9b4f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq"
Oct 30 00:59:52.479008 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.478988    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c60cba8a-34d4-454c-8405-9afa8c9b4f76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-601aea-nzcvq\" (UID: \"c60cba8a-34d4-454c-8405-9afa8c9b4f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq"
Oct 30 00:59:52.499602 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.499574    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtvkj\" (UniqueName: \"kubernetes.io/projected/c60cba8a-34d4-454c-8405-9afa8c9b4f76-kube-api-access-jtvkj\") pod \"maas-api-revoke-keys-e2e-api-b-601aea-nzcvq\" (UID: \"c60cba8a-34d4-454c-8405-9afa8c9b4f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq"
Oct 30 00:59:52.658350 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.658319    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq"
Oct 30 00:59:52.962517 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.962438    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc"]
Oct 30 00:59:52.966450 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.966424    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq"]
Oct 30 00:59:52.980328 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:52.980293    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8090513-mdxqc"]
Oct 30 00:59:53.663808 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:53.663760    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq" event={"ID":"c60cba8a-34d4-454c-8405-9afa8c9b4f76","Type":"ContainerStarted","Data":"f0f6d35d6eecd514dcc639efc33a948bf9246a7b6e63c7207e57f9e6fa408ab3"}
Oct 30 00:59:53.664198 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:53.663827    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq" event={"ID":"c60cba8a-34d4-454c-8405-9afa8c9b4f76","Type":"ContainerStarted","Data":"1ddd4fc23042575dfe8c1f86600962873b68368e6390175f37fe0bc28588ca74"}
Oct 30 00:59:53.703962 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:53.703909    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq" podStartSLOduration=1.7038913 podStartE2EDuration="1.7038913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:53.702755244 +0000 UTC m=+1487.559662999" watchObservedRunningTime="2026-10-30 00:59:53.7038913 +0000 UTC m=+1487.560799056"
Oct 30 00:59:54.264111 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:54.264080    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55372fd8-ea01-4397-be34-ab2260d9ef33" path="/var/lib/kubelet/pods/55372fd8-ea01-4397-be34-ab2260d9ef33/volumes"
Oct 30 00:59:54.674415 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:54.674380    6099 generic.go:356] "Generic (PLEG): container finished" podID="c60cba8a-34d4-454c-8405-9afa8c9b4f76" containerID="f0f6d35d6eecd514dcc639efc33a948bf9246a7b6e63c7207e57f9e6fa408ab3" exitCode=0
Oct 30 00:59:54.674833 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:54.674441    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq" event={"ID":"c60cba8a-34d4-454c-8405-9afa8c9b4f76","Type":"ContainerDied","Data":"f0f6d35d6eecd514dcc639efc33a948bf9246a7b6e63c7207e57f9e6fa408ab3"}
Oct 30 00:59:54.868045 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:54.868011    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"]
Oct 30 00:59:54.873131 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:54.873101    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"
Oct 30 00:59:54.893797 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:54.893759    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5f4f9b-serving-cert\""
Oct 30 00:59:54.901223 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:54.901200    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"]
Oct 30 00:59:54.916500 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:54.916464    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-metrics-certs\") pod \"maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb\" (UID: \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"
Oct 30 00:59:54.916644 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:54.916567    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb\" (UID: \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"
Oct 30 00:59:54.916774 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:54.916648    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-maas-api-tls\") pod \"maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb\" (UID: \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"
Oct 30 00:59:54.916774 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:54.916768    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc8lg\" (UniqueName: \"kubernetes.io/projected/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-kube-api-access-fc8lg\") pod \"maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb\" (UID: \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"
Oct 30 00:59:55.018170 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.018083    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-metrics-certs\") pod \"maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb\" (UID: \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"
Oct 30 00:59:55.018170 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.018139    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb\" (UID: \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"
Oct 30 00:59:55.018397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.018201    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-maas-api-tls\") pod \"maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb\" (UID: \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"
Oct 30 00:59:55.018397 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.018299    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc8lg\" (UniqueName: \"kubernetes.io/projected/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-kube-api-access-fc8lg\") pod \"maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb\" (UID: \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"
Oct 30 00:59:55.018976 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.018945    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb\" (UID: \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"
Oct 30 00:59:55.020688 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.020666    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-maas-api-tls\") pod \"maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb\" (UID: \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"
Oct 30 00:59:55.020817 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.020795    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-metrics-certs\") pod \"maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb\" (UID: \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"
Oct 30 00:59:55.041277 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.041248    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc8lg\" (UniqueName: \"kubernetes.io/projected/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-kube-api-access-fc8lg\") pod \"maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb\" (UID: \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"
Oct 30 00:59:55.187909 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.187878    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"
Oct 30 00:59:55.513516 ip-10-0-130-245 kubenswrapper[6099]: W1030 00:59:55.513474    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2a1b79_8e4f_4b19_9cc3_8c34d48d7e83.slice/crio-ccd21283187c09292c6d987b6c59901330c13a4406a878d331e17abb95c45ab7 WatchSource:0}: Error finding container ccd21283187c09292c6d987b6c59901330c13a4406a878d331e17abb95c45ab7: Status 404 returned error can't find the container with id ccd21283187c09292c6d987b6c59901330c13a4406a878d331e17abb95c45ab7
Oct 30 00:59:55.514435 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.514410    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"]
Oct 30 00:59:55.515073 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.515055    6099 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:59:55.683511 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.683479    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb" event={"ID":"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83","Type":"ContainerStarted","Data":"ccd21283187c09292c6d987b6c59901330c13a4406a878d331e17abb95c45ab7"}
Oct 30 00:59:55.842842 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.842796    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq"
Oct 30 00:59:55.942963 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.942882    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60cba8a-34d4-454c-8405-9afa8c9b4f76-kube-api-access-jtvkj\" (UniqueName: \"kubernetes.io/projected/c60cba8a-34d4-454c-8405-9afa8c9b4f76-kube-api-access-jtvkj\") pod \"c60cba8a-34d4-454c-8405-9afa8c9b4f76\" (UID: \"c60cba8a-34d4-454c-8405-9afa8c9b4f76\") "
Oct 30 00:59:55.943118 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.942994    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c60cba8a-34d4-454c-8405-9afa8c9b4f76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c60cba8a-34d4-454c-8405-9afa8c9b4f76-maas-api-openshift-ca\") pod \"c60cba8a-34d4-454c-8405-9afa8c9b4f76\" (UID: \"c60cba8a-34d4-454c-8405-9afa8c9b4f76\") "
Oct 30 00:59:55.943118 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.943037    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c60cba8a-34d4-454c-8405-9afa8c9b4f76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c60cba8a-34d4-454c-8405-9afa8c9b4f76-maas-api-xks-ca\") pod \"c60cba8a-34d4-454c-8405-9afa8c9b4f76\" (UID: \"c60cba8a-34d4-454c-8405-9afa8c9b4f76\") "
Oct 30 00:59:55.943648 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.943620    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60cba8a-34d4-454c-8405-9afa8c9b4f76-maas-api-openshift-ca" pod "c60cba8a-34d4-454c-8405-9afa8c9b4f76" (UID: "c60cba8a-34d4-454c-8405-9afa8c9b4f76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:55.943933 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.943905    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c60cba8a-34d4-454c-8405-9afa8c9b4f76-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:55.945795 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.945769    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c60cba8a-34d4-454c-8405-9afa8c9b4f76-maas-api-xks-ca" pod "c60cba8a-34d4-454c-8405-9afa8c9b4f76" (UID: "c60cba8a-34d4-454c-8405-9afa8c9b4f76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:59:55.953370 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:55.953336    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60cba8a-34d4-454c-8405-9afa8c9b4f76-kube-api-access-jtvkj" pod "c60cba8a-34d4-454c-8405-9afa8c9b4f76" (UID: "c60cba8a-34d4-454c-8405-9afa8c9b4f76"). InnerVolumeSpecName "kube-api-access-jtvkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:56.045332 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:56.045303    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtvkj\" (UniqueName: \"kubernetes.io/projected/c60cba8a-34d4-454c-8405-9afa8c9b4f76-kube-api-access-jtvkj\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:56.045332 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:56.045330    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c60cba8a-34d4-454c-8405-9afa8c9b4f76-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:56.693683 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:56.693655    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq"
Oct 30 00:59:56.693683 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:56.693680    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq" event={"ID":"c60cba8a-34d4-454c-8405-9afa8c9b4f76","Type":"ContainerDied","Data":"1ddd4fc23042575dfe8c1f86600962873b68368e6390175f37fe0bc28588ca74"}
Oct 30 00:59:56.694124 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:56.693738    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ddd4fc23042575dfe8c1f86600962873b68368e6390175f37fe0bc28588ca74"
Oct 30 00:59:56.695765 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:56.695698    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb" event={"ID":"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83","Type":"ContainerStarted","Data":"852dd853318a644ee6423e99e4a519aa2c276e7a1764069179f1c7b47b0002af"}
Oct 30 00:59:56.695920 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:56.695797    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"
Oct 30 00:59:56.737483 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:56.737436    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb" podStartSLOduration=2.423972497 podStartE2EDuration="2.737421631s" totalImagesPullingTime="313.449134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:56.736664529 +0000 UTC m=+1490.593572285" watchObservedRunningTime="2026-10-30 00:59:56.737421631 +0000 UTC m=+1490.594329386"
Oct 30 00:59:56.936635 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:56.936600    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:59:57.070753 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.070720    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9004633-73af-405e-8cd8-b1bd569602af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9004633-73af-405e-8cd8-b1bd569602af-istiod-ca-cert\") pod \"a9004633-73af-405e-8cd8-b1bd569602af\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") "
Oct 30 00:59:57.070938 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.070769    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-istio-data\") pod \"a9004633-73af-405e-8cd8-b1bd569602af\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") "
Oct 30 00:59:57.070994 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.070962    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9004633-73af-405e-8cd8-b1bd569602af-istio-token\" (UniqueName: \"kubernetes.io/projected/a9004633-73af-405e-8cd8-b1bd569602af-istio-token\") pod \"a9004633-73af-405e-8cd8-b1bd569602af\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") "
Oct 30 00:59:57.071087 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.071066    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-istio-data" pod "a9004633-73af-405e-8cd8-b1bd569602af" (UID: "a9004633-73af-405e-8cd8-b1bd569602af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:57.071133 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.071095    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9004633-73af-405e-8cd8-b1bd569602af-istiod-ca-cert" pod "a9004633-73af-405e-8cd8-b1bd569602af" (UID: "a9004633-73af-405e-8cd8-b1bd569602af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:57.071177 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.071136    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-istio-envoy\") pod \"a9004633-73af-405e-8cd8-b1bd569602af\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") "
Oct 30 00:59:57.071177 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.071166    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-credential-socket\") pod \"a9004633-73af-405e-8cd8-b1bd569602af\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") "
Oct 30 00:59:57.071263 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.071241    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-workload-certs\") pod \"a9004633-73af-405e-8cd8-b1bd569602af\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") "
Oct 30 00:59:57.071375 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.071357    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-workload-socket\") pod \"a9004633-73af-405e-8cd8-b1bd569602af\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") "
Oct 30 00:59:57.071482 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.071466    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9004633-73af-405e-8cd8-b1bd569602af-kube-api-access-w8x6q\" (UniqueName: \"kubernetes.io/projected/a9004633-73af-405e-8cd8-b1bd569602af-kube-api-access-w8x6q\") pod \"a9004633-73af-405e-8cd8-b1bd569602af\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") "
Oct 30 00:59:57.071554 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.071514    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-credential-socket" pod "a9004633-73af-405e-8cd8-b1bd569602af" (UID: "a9004633-73af-405e-8cd8-b1bd569602af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:57.071554 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.071532    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9004633-73af-405e-8cd8-b1bd569602af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9004633-73af-405e-8cd8-b1bd569602af-istio-podinfo\") pod \"a9004633-73af-405e-8cd8-b1bd569602af\" (UID: \"a9004633-73af-405e-8cd8-b1bd569602af\") "
Oct 30 00:59:57.072184 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.071594    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-workload-certs" pod "a9004633-73af-405e-8cd8-b1bd569602af" (UID: "a9004633-73af-405e-8cd8-b1bd569602af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:57.072184 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.071655    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-workload-socket" pod "a9004633-73af-405e-8cd8-b1bd569602af" (UID: "a9004633-73af-405e-8cd8-b1bd569602af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:57.072315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.072257    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9004633-73af-405e-8cd8-b1bd569602af-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:57.072315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.072274    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:57.072315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.072282    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:57.072315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.072291    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:57.072315 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.072300    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:57.073447 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.073401    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-istio-envoy" pod "a9004633-73af-405e-8cd8-b1bd569602af" (UID: "a9004633-73af-405e-8cd8-b1bd569602af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:57.073573 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.073541    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9004633-73af-405e-8cd8-b1bd569602af-istio-token" pod "a9004633-73af-405e-8cd8-b1bd569602af" (UID: "a9004633-73af-405e-8cd8-b1bd569602af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:57.073830 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.073794    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9004633-73af-405e-8cd8-b1bd569602af-kube-api-access-w8x6q" pod "a9004633-73af-405e-8cd8-b1bd569602af" (UID: "a9004633-73af-405e-8cd8-b1bd569602af"). InnerVolumeSpecName "kube-api-access-w8x6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:57.073938 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.073908    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9004633-73af-405e-8cd8-b1bd569602af-istio-podinfo" pod "a9004633-73af-405e-8cd8-b1bd569602af" (UID: "a9004633-73af-405e-8cd8-b1bd569602af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:57.173834 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.173751    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8x6q\" (UniqueName: \"kubernetes.io/projected/a9004633-73af-405e-8cd8-b1bd569602af-kube-api-access-w8x6q\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:57.173834 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.173785    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9004633-73af-405e-8cd8-b1bd569602af-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:57.173834 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.173796    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9004633-73af-405e-8cd8-b1bd569602af-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:57.173834 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.173804    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9004633-73af-405e-8cd8-b1bd569602af-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 00:59:57.706472 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.706436    6099 generic.go:356] "Generic (PLEG): container finished" podID="a9004633-73af-405e-8cd8-b1bd569602af" containerID="8df45b1fc6e1b6b783d73152c440ddb40e495192b42cf37333904d5061a90f36" exitCode=0
Oct 30 00:59:57.706961 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.706489    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"
Oct 30 00:59:57.706961 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.706519    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr" event={"ID":"a9004633-73af-405e-8cd8-b1bd569602af","Type":"ContainerDied","Data":"8df45b1fc6e1b6b783d73152c440ddb40e495192b42cf37333904d5061a90f36"}
Oct 30 00:59:57.706961 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.706577    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr" event={"ID":"a9004633-73af-405e-8cd8-b1bd569602af","Type":"ContainerDied","Data":"f959cfd44ae2c96b412eaf41a1731da5d7fbdb3c618f4441a2db8fe42fef2fd1"}
Oct 30 00:59:57.706961 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.706608    6099 scope.go:122] "RemoveContainer" containerID="8df45b1fc6e1b6b783d73152c440ddb40e495192b42cf37333904d5061a90f36"
Oct 30 00:59:57.719392 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.719371    6099 scope.go:122] "RemoveContainer" containerID="c843d64a739e4001754f1931db315e4e062563754e08d3d9a1b14440f895b27d"
Oct 30 00:59:57.730317 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.730148    6099 scope.go:122] "RemoveContainer" containerID="8df45b1fc6e1b6b783d73152c440ddb40e495192b42cf37333904d5061a90f36"
Oct 30 00:59:57.730438 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:59:57.730419    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df45b1fc6e1b6b783d73152c440ddb40e495192b42cf37333904d5061a90f36\": container with ID starting with 8df45b1fc6e1b6b783d73152c440ddb40e495192b42cf37333904d5061a90f36 not found: ID does not exist" containerID="8df45b1fc6e1b6b783d73152c440ddb40e495192b42cf37333904d5061a90f36"
Oct 30 00:59:57.730489 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.730447    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df45b1fc6e1b6b783d73152c440ddb40e495192b42cf37333904d5061a90f36"} err="failed to get container status \"8df45b1fc6e1b6b783d73152c440ddb40e495192b42cf37333904d5061a90f36\": rpc error: code = NotFound desc = could not find container \"8df45b1fc6e1b6b783d73152c440ddb40e495192b42cf37333904d5061a90f36\": container with ID starting with 8df45b1fc6e1b6b783d73152c440ddb40e495192b42cf37333904d5061a90f36 not found: ID does not exist"
Oct 30 00:59:57.730489 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.730466    6099 scope.go:122] "RemoveContainer" containerID="c843d64a739e4001754f1931db315e4e062563754e08d3d9a1b14440f895b27d"
Oct 30 00:59:57.730674 ip-10-0-130-245 kubenswrapper[6099]: E1030 00:59:57.730658    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c843d64a739e4001754f1931db315e4e062563754e08d3d9a1b14440f895b27d\": container with ID starting with c843d64a739e4001754f1931db315e4e062563754e08d3d9a1b14440f895b27d not found: ID does not exist" containerID="c843d64a739e4001754f1931db315e4e062563754e08d3d9a1b14440f895b27d"
Oct 30 00:59:57.730748 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.730677    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c843d64a739e4001754f1931db315e4e062563754e08d3d9a1b14440f895b27d"} err="failed to get container status \"c843d64a739e4001754f1931db315e4e062563754e08d3d9a1b14440f895b27d\": rpc error: code = NotFound desc = could not find container \"c843d64a739e4001754f1931db315e4e062563754e08d3d9a1b14440f895b27d\": container with ID starting with c843d64a739e4001754f1931db315e4e062563754e08d3d9a1b14440f895b27d not found: ID does not exist"
Oct 30 00:59:57.759990 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.759967    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"]
Oct 30 00:59:57.771837 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:57.771812    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-610065-openshift-default-6cbfd66d77-bs5jr"]
Oct 30 00:59:58.263527 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:58.263494    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9004633-73af-405e-8cd8-b1bd569602af" path="/var/lib/kubelet/pods/a9004633-73af-405e-8cd8-b1bd569602af/volumes"
Oct 30 00:59:59.284304 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.284262    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"]
Oct 30 00:59:59.284797 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.284514    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px" podUID="4a920d38-7878-4299-b399-64f3f76d7e55" containerName="istio-proxy" containerID="cri-o://9e0a6e0513f77dadd687fcfba21a0822533c4e25c145b08ae501f2fde03d66d0" gracePeriod=30
Oct 30 00:59:59.297796 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.297772    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"]
Oct 30 00:59:59.298010 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.297982    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj" podUID="e82201b7-f285-4ec3-ba2c-ddb29d770391" containerName="istio-proxy" containerID="cri-o://11eb2ba27d3e81ac698a5c7f261a5db3a0f6b163ca92c8f94a03719edb61dba0" gracePeriod=30
Oct 30 00:59:59.826472 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.826439    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn"]
Oct 30 00:59:59.827193 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.827176    6099 cpu_manager.go:414] "containerMap: removing container" podUID="c60cba8a-34d4-454c-8405-9afa8c9b4f76" containerName="revoke-keys"
Oct 30 00:59:59.827193 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.827192    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60cba8a-34d4-454c-8405-9afa8c9b4f76" containerName="revoke-keys"
Oct 30 00:59:59.827322 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.827244    6099 cpu_manager.go:414] "containerMap: removing container" podUID="a9004633-73af-405e-8cd8-b1bd569602af" containerName="istio-proxy"
Oct 30 00:59:59.827322 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.827249    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9004633-73af-405e-8cd8-b1bd569602af" containerName="istio-proxy"
Oct 30 00:59:59.827322 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.827257    6099 cpu_manager.go:414] "containerMap: removing container" podUID="a9004633-73af-405e-8cd8-b1bd569602af" containerName="istio-proxy"
Oct 30 00:59:59.827322 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.827261    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9004633-73af-405e-8cd8-b1bd569602af" containerName="istio-proxy"
Oct 30 00:59:59.827444 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.827371    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9004633-73af-405e-8cd8-b1bd569602af" containerName="istio-proxy"
Oct 30 00:59:59.827444 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.827379    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9004633-73af-405e-8cd8-b1bd569602af" containerName="istio-proxy"
Oct 30 00:59:59.827444 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.827388    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c60cba8a-34d4-454c-8405-9afa8c9b4f76" containerName="revoke-keys"
Oct 30 00:59:59.832457 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.832440    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn"
Oct 30 00:59:59.851298 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.851267    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn"]
Oct 30 00:59:59.916954 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.916920    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:59:59.917164 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.917117    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 00:59:59.943433 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.943402    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsrwq\" (UniqueName: \"kubernetes.io/projected/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-kube-api-access-gsrwq\") pod \"maas-api-revoke-keys-e2e-api-b-a32412-w6dtn\" (UID: \"6370d09b-ecc5-4ca5-968f-d39f3bdefcde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn"
Oct 30 00:59:59.943658 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.943638    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a32412-w6dtn\" (UID: \"6370d09b-ecc5-4ca5-968f-d39f3bdefcde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn"
Oct 30 00:59:59.943880 ip-10-0-130-245 kubenswrapper[6099]: I1030 00:59:59.943862    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a32412-w6dtn\" (UID: \"6370d09b-ecc5-4ca5-968f-d39f3bdefcde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn"
Oct 30 01:00:00.045349 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.045310    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a32412-w6dtn\" (UID: \"6370d09b-ecc5-4ca5-968f-d39f3bdefcde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn"
Oct 30 01:00:00.045545 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.045399    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsrwq\" (UniqueName: \"kubernetes.io/projected/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-kube-api-access-gsrwq\") pod \"maas-api-revoke-keys-e2e-api-b-a32412-w6dtn\" (UID: \"6370d09b-ecc5-4ca5-968f-d39f3bdefcde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn"
Oct 30 01:00:00.045649 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.045632    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a32412-w6dtn\" (UID: \"6370d09b-ecc5-4ca5-968f-d39f3bdefcde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn"
Oct 30 01:00:00.046523 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.046489    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a32412-w6dtn\" (UID: \"6370d09b-ecc5-4ca5-968f-d39f3bdefcde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn"
Oct 30 01:00:00.048814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.048791    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a32412-w6dtn\" (UID: \"6370d09b-ecc5-4ca5-968f-d39f3bdefcde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn"
Oct 30 01:00:00.068206 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.068178    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsrwq\" (UniqueName: \"kubernetes.io/projected/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-kube-api-access-gsrwq\") pod \"maas-api-revoke-keys-e2e-api-b-a32412-w6dtn\" (UID: \"6370d09b-ecc5-4ca5-968f-d39f3bdefcde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn"
Oct 30 01:00:00.152155 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.152047    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn"
Oct 30 01:00:00.155351 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.155323    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz"]
Oct 30 01:00:00.160817 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.160790    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz"
Oct 30 01:00:00.164432 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.164409    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl"]
Oct 30 01:00:00.169173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.169146    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl"
Oct 30 01:00:00.169652 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.169631    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k"]
Oct 30 01:00:00.174159 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.174141    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k"
Oct 30 01:00:00.194033 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.194006    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc"]
Oct 30 01:00:00.200667 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.200639    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz"]
Oct 30 01:00:00.200846 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.200771    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc"
Oct 30 01:00:00.213961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.210314    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2"]
Oct 30 01:00:00.267123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.266498    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzhk4\" (UniqueName: \"kubernetes.io/projected/e23eea04-ac89-4fa4-bb29-f38d655df0d7-kube-api-access-mzhk4\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k\" (UID: \"e23eea04-ac89-4fa4-bb29-f38d655df0d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k"
Oct 30 01:00:00.267123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.266647    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d10ae-6748-461e-92c4-f5efc2f0c880-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc\" (UID: \"ed2d10ae-6748-461e-92c4-f5efc2f0c880\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc"
Oct 30 01:00:00.267123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.266748    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfk5t\" (UniqueName: \"kubernetes.io/projected/ed2d10ae-6748-461e-92c4-f5efc2f0c880-kube-api-access-hfk5t\") pod \"maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc\" (UID: \"ed2d10ae-6748-461e-92c4-f5efc2f0c880\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc"
Oct 30 01:00:00.267123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.266941    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bf6w\" (UniqueName: \"kubernetes.io/projected/3365bf0b-a96c-429e-8565-faa44fef8ee0-kube-api-access-6bf6w\") pod \"maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl\" (UID: \"3365bf0b-a96c-429e-8565-faa44fef8ee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl"
Oct 30 01:00:00.267123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.266973    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3365bf0b-a96c-429e-8565-faa44fef8ee0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl\" (UID: \"3365bf0b-a96c-429e-8565-faa44fef8ee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl"
Oct 30 01:00:00.267123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.267037    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23eea04-ac89-4fa4-bb29-f38d655df0d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k\" (UID: \"e23eea04-ac89-4fa4-bb29-f38d655df0d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k"
Oct 30 01:00:00.301525 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.288040    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2"
Oct 30 01:00:00.308853 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.308824    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl"]
Oct 30 01:00:00.309227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.309210    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k"]
Oct 30 01:00:00.309366 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.309355    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc"]
Oct 30 01:00:00.309491 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.309479    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2"]
Oct 30 01:00:00.309603 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.309593    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl"]
Oct 30 01:00:00.316609 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.316591    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw"]
Oct 30 01:00:00.317142 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.317121    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl"
Oct 30 01:00:00.322796 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.322775    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7"]
Oct 30 01:00:00.327354 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.327216    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw"
Oct 30 01:00:00.329583 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.329560    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl"]
Oct 30 01:00:00.329690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.329605    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw"]
Oct 30 01:00:00.329690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.329681    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7"
Oct 30 01:00:00.330539 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.330515    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7"]
Oct 30 01:00:00.367007 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.363456    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw"]
Oct 30 01:00:00.369129 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.369094    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bf6w\" (UniqueName: \"kubernetes.io/projected/3365bf0b-a96c-429e-8565-faa44fef8ee0-kube-api-access-6bf6w\") pod \"maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl\" (UID: \"3365bf0b-a96c-429e-8565-faa44fef8ee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl"
Oct 30 01:00:00.369264 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.369157    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3365bf0b-a96c-429e-8565-faa44fef8ee0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl\" (UID: \"3365bf0b-a96c-429e-8565-faa44fef8ee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl"
Oct 30 01:00:00.369264 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.369192    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2\" (UID: \"fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2"
Oct 30 01:00:00.369264 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.369197    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw"]
Oct 30 01:00:00.369264 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.369243    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23eea04-ac89-4fa4-bb29-f38d655df0d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k\" (UID: \"e23eea04-ac89-4fa4-bb29-f38d655df0d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k"
Oct 30 01:00:00.369537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.369273    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87z9f\" (UniqueName: \"kubernetes.io/projected/0e2294af-beb7-4d49-8cd8-4abd0c04a40a-kube-api-access-87z9f\") pod \"maas-api-key-cleanup-29888700-9p4dl\" (UID: \"0e2294af-beb7-4d49-8cd8-4abd0c04a40a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl"
Oct 30 01:00:00.369537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.369322    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e2294af-beb7-4d49-8cd8-4abd0c04a40a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888700-9p4dl\" (UID: \"0e2294af-beb7-4d49-8cd8-4abd0c04a40a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl"
Oct 30 01:00:00.369537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.369328    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw"
Oct 30 01:00:00.371131 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.369825    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzhk4\" (UniqueName: \"kubernetes.io/projected/e23eea04-ac89-4fa4-bb29-f38d655df0d7-kube-api-access-mzhk4\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k\" (UID: \"e23eea04-ac89-4fa4-bb29-f38d655df0d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k"
Oct 30 01:00:00.371131 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.369927    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c86d85f-10ed-4300-98d7-e6934451d12d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw\" (UID: \"9c86d85f-10ed-4300-98d7-e6934451d12d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw"
Oct 30 01:00:00.371131 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.370014    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm96k\" (UniqueName: \"kubernetes.io/projected/f378df0c-bde8-4326-ade3-7c4ecd956cb3-kube-api-access-bm96k\") pod \"maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz\" (UID: \"f378df0c-bde8-4326-ade3-7c4ecd956cb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz"
Oct 30 01:00:00.371131 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.370052    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d10ae-6748-461e-92c4-f5efc2f0c880-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc\" (UID: \"ed2d10ae-6748-461e-92c4-f5efc2f0c880\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc"
Oct 30 01:00:00.371131 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.370163    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfk5t\" (UniqueName: \"kubernetes.io/projected/ed2d10ae-6748-461e-92c4-f5efc2f0c880-kube-api-access-hfk5t\") pod \"maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc\" (UID: \"ed2d10ae-6748-461e-92c4-f5efc2f0c880\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc"
Oct 30 01:00:00.371131 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.370419    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3365bf0b-a96c-429e-8565-faa44fef8ee0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl\" (UID: \"3365bf0b-a96c-429e-8565-faa44fef8ee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl"
Oct 30 01:00:00.371131 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.371062    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23eea04-ac89-4fa4-bb29-f38d655df0d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k\" (UID: \"e23eea04-ac89-4fa4-bb29-f38d655df0d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k"
Oct 30 01:00:00.371558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.371485    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f378df0c-bde8-4326-ade3-7c4ecd956cb3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz\" (UID: \"f378df0c-bde8-4326-ade3-7c4ecd956cb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz"
Oct 30 01:00:00.371558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.371522    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ntrs\" (UniqueName: \"kubernetes.io/projected/09a29068-f886-4f57-9eab-db7680a24e7a-kube-api-access-7ntrs\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7\" (UID: \"09a29068-f886-4f57-9eab-db7680a24e7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7"
Oct 30 01:00:00.371672 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.371596    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw2pm\" (UniqueName: \"kubernetes.io/projected/fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9-kube-api-access-tw2pm\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2\" (UID: \"fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2"
Oct 30 01:00:00.371672 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.371640    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prdh9\" (UniqueName: \"kubernetes.io/projected/9c86d85f-10ed-4300-98d7-e6934451d12d-kube-api-access-prdh9\") pod \"maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw\" (UID: \"9c86d85f-10ed-4300-98d7-e6934451d12d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw"
Oct 30 01:00:00.371804 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.371678    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a29068-f886-4f57-9eab-db7680a24e7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7\" (UID: \"09a29068-f886-4f57-9eab-db7680a24e7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7"
Oct 30 01:00:00.372314 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.372196    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d10ae-6748-461e-92c4-f5efc2f0c880-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc\" (UID: \"ed2d10ae-6748-461e-92c4-f5efc2f0c880\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc"
Oct 30 01:00:00.398838 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.398807    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfk5t\" (UniqueName: \"kubernetes.io/projected/ed2d10ae-6748-461e-92c4-f5efc2f0c880-kube-api-access-hfk5t\") pod \"maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc\" (UID: \"ed2d10ae-6748-461e-92c4-f5efc2f0c880\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc"
Oct 30 01:00:00.399019 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.399000    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bf6w\" (UniqueName: \"kubernetes.io/projected/3365bf0b-a96c-429e-8565-faa44fef8ee0-kube-api-access-6bf6w\") pod \"maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl\" (UID: \"3365bf0b-a96c-429e-8565-faa44fef8ee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl"
Oct 30 01:00:00.399306 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.399288    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzhk4\" (UniqueName: \"kubernetes.io/projected/e23eea04-ac89-4fa4-bb29-f38d655df0d7-kube-api-access-mzhk4\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k\" (UID: \"e23eea04-ac89-4fa4-bb29-f38d655df0d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k"
Oct 30 01:00:00.473376 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.473263    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2\" (UID: \"fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2"
Oct 30 01:00:00.473376 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.473350    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87z9f\" (UniqueName: \"kubernetes.io/projected/0e2294af-beb7-4d49-8cd8-4abd0c04a40a-kube-api-access-87z9f\") pod \"maas-api-key-cleanup-29888700-9p4dl\" (UID: \"0e2294af-beb7-4d49-8cd8-4abd0c04a40a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl"
Oct 30 01:00:00.473589 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.473401    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e2294af-beb7-4d49-8cd8-4abd0c04a40a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888700-9p4dl\" (UID: \"0e2294af-beb7-4d49-8cd8-4abd0c04a40a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl"
Oct 30 01:00:00.473589 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.473458    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020ef144-858f-4125-a9b8-f18fbebbffad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw\" (UID: \"020ef144-858f-4125-a9b8-f18fbebbffad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw"
Oct 30 01:00:00.473589 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.473529    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c86d85f-10ed-4300-98d7-e6934451d12d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw\" (UID: \"9c86d85f-10ed-4300-98d7-e6934451d12d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw"
Oct 30 01:00:00.473740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.473601    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm96k\" (UniqueName: \"kubernetes.io/projected/f378df0c-bde8-4326-ade3-7c4ecd956cb3-kube-api-access-bm96k\") pod \"maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz\" (UID: \"f378df0c-bde8-4326-ade3-7c4ecd956cb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz"
Oct 30 01:00:00.474047 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.474020    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2\" (UID: \"fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2"
Oct 30 01:00:00.474170 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.474023    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f378df0c-bde8-4326-ade3-7c4ecd956cb3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz\" (UID: \"f378df0c-bde8-4326-ade3-7c4ecd956cb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz"
Oct 30 01:00:00.474170 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.474083    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ntrs\" (UniqueName: \"kubernetes.io/projected/09a29068-f886-4f57-9eab-db7680a24e7a-kube-api-access-7ntrs\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7\" (UID: \"09a29068-f886-4f57-9eab-db7680a24e7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7"
Oct 30 01:00:00.474170 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.474120    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9cr\" (UniqueName: \"kubernetes.io/projected/020ef144-858f-4125-a9b8-f18fbebbffad-kube-api-access-cq9cr\") pod \"maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw\" (UID: \"020ef144-858f-4125-a9b8-f18fbebbffad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw"
Oct 30 01:00:00.474170 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.474150    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw2pm\" (UniqueName: \"kubernetes.io/projected/fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9-kube-api-access-tw2pm\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2\" (UID: \"fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2"
Oct 30 01:00:00.474401 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.474181    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prdh9\" (UniqueName: \"kubernetes.io/projected/9c86d85f-10ed-4300-98d7-e6934451d12d-kube-api-access-prdh9\") pod \"maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw\" (UID: \"9c86d85f-10ed-4300-98d7-e6934451d12d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw"
Oct 30 01:00:00.474401 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.474224    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e2294af-beb7-4d49-8cd8-4abd0c04a40a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888700-9p4dl\" (UID: \"0e2294af-beb7-4d49-8cd8-4abd0c04a40a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl"
Oct 30 01:00:00.474401 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.474225    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a29068-f886-4f57-9eab-db7680a24e7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7\" (UID: \"09a29068-f886-4f57-9eab-db7680a24e7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7"
Oct 30 01:00:00.474401 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.474353    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c86d85f-10ed-4300-98d7-e6934451d12d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw\" (UID: \"9c86d85f-10ed-4300-98d7-e6934451d12d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw"
Oct 30 01:00:00.474762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.474689    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f378df0c-bde8-4326-ade3-7c4ecd956cb3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz\" (UID: \"f378df0c-bde8-4326-ade3-7c4ecd956cb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz"
Oct 30 01:00:00.475074 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.475048    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a29068-f886-4f57-9eab-db7680a24e7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7\" (UID: \"09a29068-f886-4f57-9eab-db7680a24e7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7"
Oct 30 01:00:00.499302 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.499273    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ntrs\" (UniqueName: \"kubernetes.io/projected/09a29068-f886-4f57-9eab-db7680a24e7a-kube-api-access-7ntrs\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7\" (UID: \"09a29068-f886-4f57-9eab-db7680a24e7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7"
Oct 30 01:00:00.499430 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.499307    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw2pm\" (UniqueName: \"kubernetes.io/projected/fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9-kube-api-access-tw2pm\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2\" (UID: \"fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2"
Oct 30 01:00:00.499527 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.499508    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prdh9\" (UniqueName: \"kubernetes.io/projected/9c86d85f-10ed-4300-98d7-e6934451d12d-kube-api-access-prdh9\") pod \"maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw\" (UID: \"9c86d85f-10ed-4300-98d7-e6934451d12d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw"
Oct 30 01:00:00.499608 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.499589    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm96k\" (UniqueName: \"kubernetes.io/projected/f378df0c-bde8-4326-ade3-7c4ecd956cb3-kube-api-access-bm96k\") pod \"maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz\" (UID: \"f378df0c-bde8-4326-ade3-7c4ecd956cb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz"
Oct 30 01:00:00.503125 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.503101    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn"]
Oct 30 01:00:00.503228 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:00.503213    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6370d09b_ecc5_4ca5_968f_d39f3bdefcde.slice/crio-cb369daaf7f6d7e4cc80512d4db894886657a4fc5c54a71e4be09d3374996677 WatchSource:0}: Error finding container cb369daaf7f6d7e4cc80512d4db894886657a4fc5c54a71e4be09d3374996677: Status 404 returned error can't find the container with id cb369daaf7f6d7e4cc80512d4db894886657a4fc5c54a71e4be09d3374996677
Oct 30 01:00:00.503761 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.503740    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87z9f\" (UniqueName: \"kubernetes.io/projected/0e2294af-beb7-4d49-8cd8-4abd0c04a40a-kube-api-access-87z9f\") pod \"maas-api-key-cleanup-29888700-9p4dl\" (UID: \"0e2294af-beb7-4d49-8cd8-4abd0c04a40a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl"
Oct 30 01:00:00.523212 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.523185    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz"
Oct 30 01:00:00.568677 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.568649    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl"
Oct 30 01:00:00.575698 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.575670    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020ef144-858f-4125-a9b8-f18fbebbffad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw\" (UID: \"020ef144-858f-4125-a9b8-f18fbebbffad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw"
Oct 30 01:00:00.575848 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.575826    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9cr\" (UniqueName: \"kubernetes.io/projected/020ef144-858f-4125-a9b8-f18fbebbffad-kube-api-access-cq9cr\") pod \"maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw\" (UID: \"020ef144-858f-4125-a9b8-f18fbebbffad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw"
Oct 30 01:00:00.576417 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.576383    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020ef144-858f-4125-a9b8-f18fbebbffad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw\" (UID: \"020ef144-858f-4125-a9b8-f18fbebbffad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw"
Oct 30 01:00:00.603543 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.603349    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9cr\" (UniqueName: \"kubernetes.io/projected/020ef144-858f-4125-a9b8-f18fbebbffad-kube-api-access-cq9cr\") pod \"maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw\" (UID: \"020ef144-858f-4125-a9b8-f18fbebbffad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw"
Oct 30 01:00:00.613044 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.612826    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k"
Oct 30 01:00:00.630586 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.630425    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc"
Oct 30 01:00:00.663356 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.663166    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2"
Oct 30 01:00:00.674478 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.672572    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl"
Oct 30 01:00:00.690664 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.690391    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw"
Oct 30 01:00:00.697639 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.697571    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7"
Oct 30 01:00:00.703636 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.703551    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw"
Oct 30 01:00:00.848321 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.841186    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz"]
Oct 30 01:00:00.899964 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.899928    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn" event={"ID":"6370d09b-ecc5-4ca5-968f-d39f3bdefcde","Type":"ContainerStarted","Data":"cb369daaf7f6d7e4cc80512d4db894886657a4fc5c54a71e4be09d3374996677"}
Oct 30 01:00:00.925502 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.925398    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl"]
Oct 30 01:00:00.958914 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:00.958316    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k"]
Oct 30 01:00:01.183882 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:01.183851    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a29068_f886_4f57_9eab_db7680a24e7a.slice/crio-57dc81b6fce7b754892dea3aa94e5bf9b2ff5bfbb8e4da6c474067fec287f0e9 WatchSource:0}: Error finding container 57dc81b6fce7b754892dea3aa94e5bf9b2ff5bfbb8e4da6c474067fec287f0e9: Status 404 returned error can't find the container with id 57dc81b6fce7b754892dea3aa94e5bf9b2ff5bfbb8e4da6c474067fec287f0e9
Oct 30 01:00:01.184677 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.184655    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7"]
Oct 30 01:00:01.203118 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.203088    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc"]
Oct 30 01:00:01.204506 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:01.204483    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2d10ae_6748_461e_92c4_f5efc2f0c880.slice/crio-7ec6e1bd037a065dcc5827dc82180163ad74b021b64cf82f9d57030c50af97fd WatchSource:0}: Error finding container 7ec6e1bd037a065dcc5827dc82180163ad74b021b64cf82f9d57030c50af97fd: Status 404 returned error can't find the container with id 7ec6e1bd037a065dcc5827dc82180163ad74b021b64cf82f9d57030c50af97fd
Oct 30 01:00:01.217389 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.217364    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw"]
Oct 30 01:00:01.226810 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:01.226784    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020ef144_858f_4125_a9b8_f18fbebbffad.slice/crio-09bcb6a81bb59fc4ead064544b7458082e2905ed315b14d0739140772d5e7d25 WatchSource:0}: Error finding container 09bcb6a81bb59fc4ead064544b7458082e2905ed315b14d0739140772d5e7d25: Status 404 returned error can't find the container with id 09bcb6a81bb59fc4ead064544b7458082e2905ed315b14d0739140772d5e7d25
Oct 30 01:00:01.247520 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:01.247493    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c86d85f_10ed_4300_98d7_e6934451d12d.slice/crio-e4e6ed088f5ec9dc2ecb687189b416b871ce8dc9b16d0fa23478ef614ab88e7e WatchSource:0}: Error finding container e4e6ed088f5ec9dc2ecb687189b416b871ce8dc9b16d0fa23478ef614ab88e7e: Status 404 returned error can't find the container with id e4e6ed088f5ec9dc2ecb687189b416b871ce8dc9b16d0fa23478ef614ab88e7e
Oct 30 01:00:01.247733 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.247693    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw"]
Oct 30 01:00:01.254642 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:01.254601    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2294af_beb7_4d49_8cd8_4abd0c04a40a.slice/crio-2a3c7b7be0184413ab4e2b3e2dfbe9c2b8b31adcd1c8d4b03fa98a9025caf44b WatchSource:0}: Error finding container 2a3c7b7be0184413ab4e2b3e2dfbe9c2b8b31adcd1c8d4b03fa98a9025caf44b: Status 404 returned error can't find the container with id 2a3c7b7be0184413ab4e2b3e2dfbe9c2b8b31adcd1c8d4b03fa98a9025caf44b
Oct 30 01:00:01.256757 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.256736    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl"]
Oct 30 01:00:01.270438 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.270412    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2"]
Oct 30 01:00:01.287110 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:01.287078    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1e0fff_0f9f_4dfa_89e8_c8b28ae9cbc9.slice/crio-0064b32ad697c8721ee878aac548eac71a4ea807b6c41b3064d955054a8a9ac4 WatchSource:0}: Error finding container 0064b32ad697c8721ee878aac548eac71a4ea807b6c41b3064d955054a8a9ac4: Status 404 returned error can't find the container with id 0064b32ad697c8721ee878aac548eac71a4ea807b6c41b3064d955054a8a9ac4
Oct 30 01:00:01.910776 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.910737    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl" event={"ID":"0e2294af-beb7-4d49-8cd8-4abd0c04a40a","Type":"ContainerStarted","Data":"6a271fc2164e1dadaf2a53dfa3823ba0790e1554ef64adb5b1f3adb6ed0442a1"}
Oct 30 01:00:01.911135 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.910805    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl" event={"ID":"0e2294af-beb7-4d49-8cd8-4abd0c04a40a","Type":"ContainerStarted","Data":"2a3c7b7be0184413ab4e2b3e2dfbe9c2b8b31adcd1c8d4b03fa98a9025caf44b"}
Oct 30 01:00:01.912545 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.912522    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw" event={"ID":"020ef144-858f-4125-a9b8-f18fbebbffad","Type":"ContainerStarted","Data":"44d3f0bc3f3bacb719855b5c4621dec4588bb9ca0d1068987c8c16be4a634efa"}
Oct 30 01:00:01.912662 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.912561    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw" event={"ID":"020ef144-858f-4125-a9b8-f18fbebbffad","Type":"ContainerStarted","Data":"09bcb6a81bb59fc4ead064544b7458082e2905ed315b14d0739140772d5e7d25"}
Oct 30 01:00:01.914185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.914162    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl" event={"ID":"3365bf0b-a96c-429e-8565-faa44fef8ee0","Type":"ContainerStarted","Data":"bacd5dc3467e6d19a57df0e574ce5e9564f71f5f6640167b243b55a70c052b8a"}
Oct 30 01:00:01.914275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.914204    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl" event={"ID":"3365bf0b-a96c-429e-8565-faa44fef8ee0","Type":"ContainerStarted","Data":"2cf80dc01dcee76806582442e72d51c4a102ec44f94a03eb64e9df174f648544"}
Oct 30 01:00:01.915854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.915832    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc" event={"ID":"ed2d10ae-6748-461e-92c4-f5efc2f0c880","Type":"ContainerStarted","Data":"3eb15df51a35808a7824a60fb5d144079b209ba84280af8dd9f1a8d6904a9b16"}
Oct 30 01:00:01.915955 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.915876    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc" event={"ID":"ed2d10ae-6748-461e-92c4-f5efc2f0c880","Type":"ContainerStarted","Data":"7ec6e1bd037a065dcc5827dc82180163ad74b021b64cf82f9d57030c50af97fd"}
Oct 30 01:00:01.917339 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.917315    6099 generic.go:356] "Generic (PLEG): container finished" podID="6370d09b-ecc5-4ca5-968f-d39f3bdefcde" containerID="37719d474c5121bf1ea1a50bd06a617d6923f043cfadc32cf496938fffe1f934" exitCode=0
Oct 30 01:00:01.917435 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.917389    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn" event={"ID":"6370d09b-ecc5-4ca5-968f-d39f3bdefcde","Type":"ContainerDied","Data":"37719d474c5121bf1ea1a50bd06a617d6923f043cfadc32cf496938fffe1f934"}
Oct 30 01:00:01.918958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.918939    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz" event={"ID":"f378df0c-bde8-4326-ade3-7c4ecd956cb3","Type":"ContainerStarted","Data":"31bb6b5d454f4852c3298c03a9a564a2e8406066486608ea63d1cc1df7b16d00"}
Oct 30 01:00:01.919036 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.918978    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz" event={"ID":"f378df0c-bde8-4326-ade3-7c4ecd956cb3","Type":"ContainerStarted","Data":"10c90a027f14e4d2c658fc9577c0bb6a5ab2b72598fbcd6c3073453da8e7b434"}
Oct 30 01:00:01.920293 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.920274    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2" event={"ID":"fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9","Type":"ContainerStarted","Data":"0064b32ad697c8721ee878aac548eac71a4ea807b6c41b3064d955054a8a9ac4"}
Oct 30 01:00:01.921862 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.921843    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw" event={"ID":"9c86d85f-10ed-4300-98d7-e6934451d12d","Type":"ContainerStarted","Data":"fabe165d9cb30a76572df1442604e4eb72759d6f4d1d1e873571549887cb2bd7"}
Oct 30 01:00:01.921950 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.921878    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw" event={"ID":"9c86d85f-10ed-4300-98d7-e6934451d12d","Type":"ContainerStarted","Data":"e4e6ed088f5ec9dc2ecb687189b416b871ce8dc9b16d0fa23478ef614ab88e7e"}
Oct 30 01:00:01.923116 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.923090    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7" event={"ID":"09a29068-f886-4f57-9eab-db7680a24e7a","Type":"ContainerStarted","Data":"57dc81b6fce7b754892dea3aa94e5bf9b2ff5bfbb8e4da6c474067fec287f0e9"}
Oct 30 01:00:01.924596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.924574    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k" event={"ID":"e23eea04-ac89-4fa4-bb29-f38d655df0d7","Type":"ContainerStarted","Data":"80732e41ca073ff38f177eb1806ebabdb81a9ff6737b81104d0922aee9cd70ec"}
Oct 30 01:00:01.924680 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.924614    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k" event={"ID":"e23eea04-ac89-4fa4-bb29-f38d655df0d7","Type":"ContainerStarted","Data":"172a1f527f7cdcc510552d66fb56c811d47f82c4508bad40f7ad1aed46503ced"}
Oct 30 01:00:01.952352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.952310    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl" podStartSLOduration=1.952297899 podStartE2EDuration="1.952297899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:01.948869704 +0000 UTC m=+1495.805777469" watchObservedRunningTime="2026-10-30 01:00:01.952297899 +0000 UTC m=+1495.809205651"
Oct 30 01:00:01.987648 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:01.987596    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw" podStartSLOduration=1.987582524 podStartE2EDuration="1.987582524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:01.986880162 +0000 UTC m=+1495.843787938" watchObservedRunningTime="2026-10-30 01:00:01.987582524 +0000 UTC m=+1495.844490306"
Oct 30 01:00:02.026021 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.025797    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz" podStartSLOduration=2.025780627 podStartE2EDuration="2.025780627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:02.025179217 +0000 UTC m=+1495.882086971" watchObservedRunningTime="2026-10-30 01:00:02.025780627 +0000 UTC m=+1495.882688382"
Oct 30 01:00:02.064240 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.064191    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw" podStartSLOduration=2.0641778 podStartE2EDuration="2.0641778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:02.063208108 +0000 UTC m=+1495.920115863" watchObservedRunningTime="2026-10-30 01:00:02.0641778 +0000 UTC m=+1495.921085554"
Oct 30 01:00:02.104188 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.104135    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl" podStartSLOduration=2.104122279 podStartE2EDuration="2.104122279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:02.101240919 +0000 UTC m=+1495.958148674" watchObservedRunningTime="2026-10-30 01:00:02.104122279 +0000 UTC m=+1495.961030034"
Oct 30 01:00:02.183553 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.183504    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc" podStartSLOduration=2.183491325 podStartE2EDuration="2.183491325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:02.17999172 +0000 UTC m=+1496.036899477" watchObservedRunningTime="2026-10-30 01:00:02.183491325 +0000 UTC m=+1496.040399105"
Oct 30 01:00:02.218728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.218654    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k" podStartSLOduration=2.21863962 podStartE2EDuration="2.21863962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:02.217765359 +0000 UTC m=+1496.074673114" watchObservedRunningTime="2026-10-30 01:00:02.21863962 +0000 UTC m=+1496.075547375"
Oct 30 01:00:02.712228 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.712202    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"
Oct 30 01:00:02.829505 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.829468    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg"]
Oct 30 01:00:02.863175 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.863134    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq"]
Oct 30 01:00:02.945238 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.944800    6099 generic.go:356] "Generic (PLEG): container finished" podID="fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9" containerID="bd3362ec4618f45cf825b644f1f34d409f52471a163ddc3a72c47513daa47d86" exitCode=60
Oct 30 01:00:02.945238 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.944904    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2" event={"ID":"fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9","Type":"ContainerDied","Data":"bd3362ec4618f45cf825b644f1f34d409f52471a163ddc3a72c47513daa47d86"}
Oct 30 01:00:02.945925 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.945386    6099 scope.go:122] "RemoveContainer" containerID="bd3362ec4618f45cf825b644f1f34d409f52471a163ddc3a72c47513daa47d86"
Oct 30 01:00:02.951603 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.951576    6099 generic.go:356] "Generic (PLEG): container finished" podID="9c86d85f-10ed-4300-98d7-e6934451d12d" containerID="fabe165d9cb30a76572df1442604e4eb72759d6f4d1d1e873571549887cb2bd7" exitCode=60
Oct 30 01:00:02.951769 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.951690    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw" event={"ID":"9c86d85f-10ed-4300-98d7-e6934451d12d","Type":"ContainerDied","Data":"fabe165d9cb30a76572df1442604e4eb72759d6f4d1d1e873571549887cb2bd7"}
Oct 30 01:00:02.952019 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.951992    6099 scope.go:122] "RemoveContainer" containerID="fabe165d9cb30a76572df1442604e4eb72759d6f4d1d1e873571549887cb2bd7"
Oct 30 01:00:02.953943 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.953919    6099 generic.go:356] "Generic (PLEG): container finished" podID="09a29068-f886-4f57-9eab-db7680a24e7a" containerID="f47ea9728c54cf2e278ef078128a700754551fb12ddeb26aca039426672aeed1" exitCode=60
Oct 30 01:00:02.954172 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.954092    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7" event={"ID":"09a29068-f886-4f57-9eab-db7680a24e7a","Type":"ContainerDied","Data":"f47ea9728c54cf2e278ef078128a700754551fb12ddeb26aca039426672aeed1"}
Oct 30 01:00:02.954297 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.954223    6099 scope.go:122] "RemoveContainer" containerID="f47ea9728c54cf2e278ef078128a700754551fb12ddeb26aca039426672aeed1"
Oct 30 01:00:02.956418 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.956397    6099 generic.go:356] "Generic (PLEG): container finished" podID="e23eea04-ac89-4fa4-bb29-f38d655df0d7" containerID="80732e41ca073ff38f177eb1806ebabdb81a9ff6737b81104d0922aee9cd70ec" exitCode=60
Oct 30 01:00:02.956638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.956475    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k" event={"ID":"e23eea04-ac89-4fa4-bb29-f38d655df0d7","Type":"ContainerDied","Data":"80732e41ca073ff38f177eb1806ebabdb81a9ff6737b81104d0922aee9cd70ec"}
Oct 30 01:00:02.956764 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.956718    6099 scope.go:122] "RemoveContainer" containerID="80732e41ca073ff38f177eb1806ebabdb81a9ff6737b81104d0922aee9cd70ec"
Oct 30 01:00:02.958984 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.958961    6099 generic.go:356] "Generic (PLEG): container finished" podID="0e2294af-beb7-4d49-8cd8-4abd0c04a40a" containerID="6a271fc2164e1dadaf2a53dfa3823ba0790e1554ef64adb5b1f3adb6ed0442a1" exitCode=60
Oct 30 01:00:02.959082 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.959053    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl" event={"ID":"0e2294af-beb7-4d49-8cd8-4abd0c04a40a","Type":"ContainerDied","Data":"6a271fc2164e1dadaf2a53dfa3823ba0790e1554ef64adb5b1f3adb6ed0442a1"}
Oct 30 01:00:02.959502 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.959247    6099 scope.go:122] "RemoveContainer" containerID="6a271fc2164e1dadaf2a53dfa3823ba0790e1554ef64adb5b1f3adb6ed0442a1"
Oct 30 01:00:02.973557 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.973532    6099 generic.go:356] "Generic (PLEG): container finished" podID="020ef144-858f-4125-a9b8-f18fbebbffad" containerID="44d3f0bc3f3bacb719855b5c4621dec4588bb9ca0d1068987c8c16be4a634efa" exitCode=60
Oct 30 01:00:02.973647 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.973629    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw" event={"ID":"020ef144-858f-4125-a9b8-f18fbebbffad","Type":"ContainerDied","Data":"44d3f0bc3f3bacb719855b5c4621dec4588bb9ca0d1068987c8c16be4a634efa"}
Oct 30 01:00:02.974087 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.974066    6099 scope.go:122] "RemoveContainer" containerID="44d3f0bc3f3bacb719855b5c4621dec4588bb9ca0d1068987c8c16be4a634efa"
Oct 30 01:00:02.976629 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.976605    6099 generic.go:356] "Generic (PLEG): container finished" podID="3365bf0b-a96c-429e-8565-faa44fef8ee0" containerID="bacd5dc3467e6d19a57df0e574ce5e9564f71f5f6640167b243b55a70c052b8a" exitCode=60
Oct 30 01:00:02.976751 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.976697    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl" event={"ID":"3365bf0b-a96c-429e-8565-faa44fef8ee0","Type":"ContainerDied","Data":"bacd5dc3467e6d19a57df0e574ce5e9564f71f5f6640167b243b55a70c052b8a"}
Oct 30 01:00:02.977093 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.977074    6099 scope.go:122] "RemoveContainer" containerID="bacd5dc3467e6d19a57df0e574ce5e9564f71f5f6640167b243b55a70c052b8a"
Oct 30 01:00:02.979511 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.979485    6099 generic.go:356] "Generic (PLEG): container finished" podID="ed2d10ae-6748-461e-92c4-f5efc2f0c880" containerID="3eb15df51a35808a7824a60fb5d144079b209ba84280af8dd9f1a8d6904a9b16" exitCode=60
Oct 30 01:00:02.979633 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.979617    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc" event={"ID":"ed2d10ae-6748-461e-92c4-f5efc2f0c880","Type":"ContainerDied","Data":"3eb15df51a35808a7824a60fb5d144079b209ba84280af8dd9f1a8d6904a9b16"}
Oct 30 01:00:02.980084 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.980064    6099 scope.go:122] "RemoveContainer" containerID="3eb15df51a35808a7824a60fb5d144079b209ba84280af8dd9f1a8d6904a9b16"
Oct 30 01:00:02.983117 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.982444    6099 generic.go:356] "Generic (PLEG): container finished" podID="f378df0c-bde8-4326-ade3-7c4ecd956cb3" containerID="31bb6b5d454f4852c3298c03a9a564a2e8406066486608ea63d1cc1df7b16d00" exitCode=60
Oct 30 01:00:02.983117 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.982648    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz" event={"ID":"f378df0c-bde8-4326-ade3-7c4ecd956cb3","Type":"ContainerDied","Data":"31bb6b5d454f4852c3298c03a9a564a2e8406066486608ea63d1cc1df7b16d00"}
Oct 30 01:00:02.983117 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:02.982672    6099 scope.go:122] "RemoveContainer" containerID="31bb6b5d454f4852c3298c03a9a564a2e8406066486608ea63d1cc1df7b16d00"
Oct 30 01:00:03.344925 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:03.344892    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn"
Oct 30 01:00:03.447014 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:03.446982    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-maas-api-openshift-ca\") pod \"6370d09b-ecc5-4ca5-968f-d39f3bdefcde\" (UID: \"6370d09b-ecc5-4ca5-968f-d39f3bdefcde\") "
Oct 30 01:00:03.447913 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:03.447889    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-kube-api-access-gsrwq\" (UniqueName: \"kubernetes.io/projected/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-kube-api-access-gsrwq\") pod \"6370d09b-ecc5-4ca5-968f-d39f3bdefcde\" (UID: \"6370d09b-ecc5-4ca5-968f-d39f3bdefcde\") "
Oct 30 01:00:03.448444 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:03.448425    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-maas-api-xks-ca\") pod \"6370d09b-ecc5-4ca5-968f-d39f3bdefcde\" (UID: \"6370d09b-ecc5-4ca5-968f-d39f3bdefcde\") "
Oct 30 01:00:03.450097 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:03.447816    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-maas-api-openshift-ca" pod "6370d09b-ecc5-4ca5-968f-d39f3bdefcde" (UID: "6370d09b-ecc5-4ca5-968f-d39f3bdefcde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:03.502340 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:03.502299    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-kube-api-access-gsrwq" pod "6370d09b-ecc5-4ca5-968f-d39f3bdefcde" (UID: "6370d09b-ecc5-4ca5-968f-d39f3bdefcde"). InnerVolumeSpecName "kube-api-access-gsrwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:03.534817 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:03.525049    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-maas-api-xks-ca" pod "6370d09b-ecc5-4ca5-968f-d39f3bdefcde" (UID: "6370d09b-ecc5-4ca5-968f-d39f3bdefcde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:03.598914 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:03.596044    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:03.598914 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:03.596079    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsrwq\" (UniqueName: \"kubernetes.io/projected/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-kube-api-access-gsrwq\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:03.598914 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:03.596117    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6370d09b-ecc5-4ca5-968f-d39f3bdefcde-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:03.996039 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:03.996018    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn"
Oct 30 01:00:03.996379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:03.996019    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn" event={"ID":"6370d09b-ecc5-4ca5-968f-d39f3bdefcde","Type":"ContainerDied","Data":"cb369daaf7f6d7e4cc80512d4db894886657a4fc5c54a71e4be09d3374996677"}
Oct 30 01:00:03.996379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:03.996174    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb369daaf7f6d7e4cc80512d4db894886657a4fc5c54a71e4be09d3374996677"
Oct 30 01:00:04.000407 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.000378    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz" event={"ID":"f378df0c-bde8-4326-ade3-7c4ecd956cb3","Type":"ContainerStarted","Data":"97bdba38dd333e0e077dc6bc4ddbad1d9eaae9e898fc56e46b79b4ecafa9c486"}
Oct 30 01:00:04.003903 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.003876    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2" event={"ID":"fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9","Type":"ContainerStarted","Data":"f20075a6f5d98bfae69954722ec529e46eb1a8d32b4b163ce4780e63dc036d85"}
Oct 30 01:00:04.006405 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.006091    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw" event={"ID":"9c86d85f-10ed-4300-98d7-e6934451d12d","Type":"ContainerStarted","Data":"2deb607e40577b597147bf69eb9084f857e641f88b8b499bb8b6e6be47e8b8db"}
Oct 30 01:00:04.008422 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.008368    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k" event={"ID":"e23eea04-ac89-4fa4-bb29-f38d655df0d7","Type":"ContainerStarted","Data":"d67ec0d59179b350dccaa52c3d47aa626b48b5c1b1d4af70bbcbd1fa8e9fd87d"}
Oct 30 01:00:04.010845 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.010820    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl" event={"ID":"0e2294af-beb7-4d49-8cd8-4abd0c04a40a","Type":"ContainerStarted","Data":"fdfe9116ab85773c9d0fdf56b577a257ad05d0c5df76babeac5f96f334495b9d"}
Oct 30 01:00:04.012683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.012661    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw" event={"ID":"020ef144-858f-4125-a9b8-f18fbebbffad","Type":"ContainerStarted","Data":"a331f8481dc3d340991fa433417ca5d071e346625bd5e76932ede083f43e4da9"}
Oct 30 01:00:04.014351 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.014322    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl" event={"ID":"3365bf0b-a96c-429e-8565-faa44fef8ee0","Type":"ContainerStarted","Data":"2f86196dc2acb9a1ca5a79d3cd75b1b22ee6257de4798d53c28e593e6855b96a"}
Oct 30 01:00:04.297812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.297749    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2" podStartSLOduration=4.297731405 podStartE2EDuration="4.297731405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:04.294368768 +0000 UTC m=+1498.151276524" watchObservedRunningTime="2026-10-30 01:00:04.297731405 +0000 UTC m=+1498.154639161"
Oct 30 01:00:04.326953 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.326923    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"]
Oct 30 01:00:04.327187 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.327162    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps" podUID="254ea03c-844c-4295-aa4b-7b57cbe7efed" containerName="maas-api" containerID="cri-o://3873b8b3b12c2553fa42e2827ace0123907174bb839eea5aa3abe71580f14304" gracePeriod=30
Oct 30 01:00:04.426207 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.426142    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw"]
Oct 30 01:00:04.802252 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.802212    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 01:00:04.913886 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.913857    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 01:00:04.919572 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.919541    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"
Oct 30 01:00:04.986446 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.986346    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-podinfo\") pod \"e82201b7-f285-4ec3-ba2c-ddb29d770391\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") "
Oct 30 01:00:04.986852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.986755    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-credential-socket\") pod \"e82201b7-f285-4ec3-ba2c-ddb29d770391\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") "
Oct 30 01:00:04.987149 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.986900    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-envoy\") pod \"e82201b7-f285-4ec3-ba2c-ddb29d770391\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") "
Oct 30 01:00:04.987149 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.987055    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-data\") pod \"e82201b7-f285-4ec3-ba2c-ddb29d770391\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") "
Oct 30 01:00:04.987672 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.987096    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e82201b7-f285-4ec3-ba2c-ddb29d770391-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e82201b7-f285-4ec3-ba2c-ddb29d770391-istiod-ca-cert\") pod \"e82201b7-f285-4ec3-ba2c-ddb29d770391\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") "
Oct 30 01:00:04.987672 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.987243    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-workload-certs\") pod \"e82201b7-f285-4ec3-ba2c-ddb29d770391\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") "
Oct 30 01:00:04.988373 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.988097    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e82201b7-f285-4ec3-ba2c-ddb29d770391-kube-api-access-nfgwx\" (UniqueName: \"kubernetes.io/projected/e82201b7-f285-4ec3-ba2c-ddb29d770391-kube-api-access-nfgwx\") pod \"e82201b7-f285-4ec3-ba2c-ddb29d770391\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") "
Oct 30 01:00:04.993781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.993648    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-token\" (UniqueName: \"kubernetes.io/projected/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-token\") pod \"e82201b7-f285-4ec3-ba2c-ddb29d770391\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") "
Oct 30 01:00:04.993781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.993689    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-workload-socket\") pod \"e82201b7-f285-4ec3-ba2c-ddb29d770391\" (UID: \"e82201b7-f285-4ec3-ba2c-ddb29d770391\") "
Oct 30 01:00:04.997726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.997504    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-workload-socket" pod "e82201b7-f285-4ec3-ba2c-ddb29d770391" (UID: "e82201b7-f285-4ec3-ba2c-ddb29d770391"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:04.998372 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.998175    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-data" pod "e82201b7-f285-4ec3-ba2c-ddb29d770391" (UID: "e82201b7-f285-4ec3-ba2c-ddb29d770391"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:04.998828 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.998594    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e82201b7-f285-4ec3-ba2c-ddb29d770391-istiod-ca-cert" pod "e82201b7-f285-4ec3-ba2c-ddb29d770391" (UID: "e82201b7-f285-4ec3-ba2c-ddb29d770391"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:04.999306 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:04.999018    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-workload-certs" pod "e82201b7-f285-4ec3-ba2c-ddb29d770391" (UID: "e82201b7-f285-4ec3-ba2c-ddb29d770391"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:05.002298 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.002278    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-credential-socket" pod "e82201b7-f285-4ec3-ba2c-ddb29d770391" (UID: "e82201b7-f285-4ec3-ba2c-ddb29d770391"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:05.011754 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.011730    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-envoy" pod "e82201b7-f285-4ec3-ba2c-ddb29d770391" (UID: "e82201b7-f285-4ec3-ba2c-ddb29d770391"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:05.012745 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.012691    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-podinfo" pod "e82201b7-f285-4ec3-ba2c-ddb29d770391" (UID: "e82201b7-f285-4ec3-ba2c-ddb29d770391"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:00:05.013153 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.013121    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-token" pod "e82201b7-f285-4ec3-ba2c-ddb29d770391" (UID: "e82201b7-f285-4ec3-ba2c-ddb29d770391"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:05.013369 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.013343    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82201b7-f285-4ec3-ba2c-ddb29d770391-kube-api-access-nfgwx" pod "e82201b7-f285-4ec3-ba2c-ddb29d770391" (UID: "e82201b7-f285-4ec3-ba2c-ddb29d770391"). InnerVolumeSpecName "kube-api-access-nfgwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:05.060557 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.060498    6099 generic.go:356] "Generic (PLEG): container finished" podID="3365bf0b-a96c-429e-8565-faa44fef8ee0" containerID="2f86196dc2acb9a1ca5a79d3cd75b1b22ee6257de4798d53c28e593e6855b96a" exitCode=60
Oct 30 01:00:05.066349 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.063461    6099 scope.go:122] "RemoveContainer" containerID="2f86196dc2acb9a1ca5a79d3cd75b1b22ee6257de4798d53c28e593e6855b96a"
Oct 30 01:00:05.067052 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:05.067002    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl_odh-ai-gateway-infra(3365bf0b-a96c-429e-8565-faa44fef8ee0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl" podUID="3365bf0b-a96c-429e-8565-faa44fef8ee0"
Oct 30 01:00:05.067416 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.063774    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl" event={"ID":"3365bf0b-a96c-429e-8565-faa44fef8ee0","Type":"ContainerDied","Data":"2f86196dc2acb9a1ca5a79d3cd75b1b22ee6257de4798d53c28e593e6855b96a"}
Oct 30 01:00:05.067416 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.067290    6099 scope.go:122] "RemoveContainer" containerID="bacd5dc3467e6d19a57df0e574ce5e9564f71f5f6640167b243b55a70c052b8a"
Oct 30 01:00:05.077848 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.076926    6099 generic.go:356] "Generic (PLEG): container finished" podID="ed2d10ae-6748-461e-92c4-f5efc2f0c880" containerID="4a1dd6cb9cf50a6d1c5442604282d5504e508652d61ad1f2a648c2ed93add066" exitCode=60
Oct 30 01:00:05.077848 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.077158    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc" event={"ID":"ed2d10ae-6748-461e-92c4-f5efc2f0c880","Type":"ContainerDied","Data":"4a1dd6cb9cf50a6d1c5442604282d5504e508652d61ad1f2a648c2ed93add066"}
Oct 30 01:00:05.085780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.085164    6099 scope.go:122] "RemoveContainer" containerID="4a1dd6cb9cf50a6d1c5442604282d5504e508652d61ad1f2a648c2ed93add066"
Oct 30 01:00:05.085780 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:05.085541    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc_odh-ai-gateway-infra(ed2d10ae-6748-461e-92c4-f5efc2f0c880)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc" podUID="ed2d10ae-6748-461e-92c4-f5efc2f0c880"
Oct 30 01:00:05.092182 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.091024    6099 generic.go:356] "Generic (PLEG): container finished" podID="f378df0c-bde8-4326-ade3-7c4ecd956cb3" containerID="97bdba38dd333e0e077dc6bc4ddbad1d9eaae9e898fc56e46b79b4ecafa9c486" exitCode=60
Oct 30 01:00:05.092182 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.091141    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz" event={"ID":"f378df0c-bde8-4326-ade3-7c4ecd956cb3","Type":"ContainerDied","Data":"97bdba38dd333e0e077dc6bc4ddbad1d9eaae9e898fc56e46b79b4ecafa9c486"}
Oct 30 01:00:05.092182 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.091641    6099 scope.go:122] "RemoveContainer" containerID="97bdba38dd333e0e077dc6bc4ddbad1d9eaae9e898fc56e46b79b4ecafa9c486"
Oct 30 01:00:05.092182 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:05.092116    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz_odh-ai-gateway-infra(f378df0c-bde8-4326-ade3-7c4ecd956cb3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz" podUID="f378df0c-bde8-4326-ade3-7c4ecd956cb3"
Oct 30 01:00:05.095558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.095385    6099 generic.go:356] "Generic (PLEG): container finished" podID="fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9" containerID="f20075a6f5d98bfae69954722ec529e46eb1a8d32b4b163ce4780e63dc036d85" exitCode=60
Oct 30 01:00:05.095838 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.095816    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2" event={"ID":"fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9","Type":"ContainerDied","Data":"f20075a6f5d98bfae69954722ec529e46eb1a8d32b4b163ce4780e63dc036d85"}
Oct 30 01:00:05.096450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.096430    6099 scope.go:122] "RemoveContainer" containerID="f20075a6f5d98bfae69954722ec529e46eb1a8d32b4b163ce4780e63dc036d85"
Oct 30 01:00:05.098210 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:05.098184    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2_odh-ai-gateway-infra(fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2" podUID="fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9"
Oct 30 01:00:05.099259 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.099232    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254ea03c-844c-4295-aa4b-7b57cbe7efed-kube-api-access-hr9pj\" (UniqueName: \"kubernetes.io/projected/254ea03c-844c-4295-aa4b-7b57cbe7efed-kube-api-access-hr9pj\") pod \"254ea03c-844c-4295-aa4b-7b57cbe7efed\" (UID: \"254ea03c-844c-4295-aa4b-7b57cbe7efed\") "
Oct 30 01:00:05.120897 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.106295    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254ea03c-844c-4295-aa4b-7b57cbe7efed-kube-api-access-hr9pj" pod "254ea03c-844c-4295-aa4b-7b57cbe7efed" (UID: "254ea03c-844c-4295-aa4b-7b57cbe7efed"). InnerVolumeSpecName "kube-api-access-hr9pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:05.121059 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.120996    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-workload-socket\") pod \"4a920d38-7878-4299-b399-64f3f76d7e55\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") "
Oct 30 01:00:05.121059 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.121043    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-istio-data\") pod \"4a920d38-7878-4299-b399-64f3f76d7e55\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") "
Oct 30 01:00:05.121339 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.121073    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a920d38-7878-4299-b399-64f3f76d7e55-istio-token\" (UniqueName: \"kubernetes.io/projected/4a920d38-7878-4299-b399-64f3f76d7e55-istio-token\") pod \"4a920d38-7878-4299-b399-64f3f76d7e55\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") "
Oct 30 01:00:05.121339 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.121161    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/254ea03c-844c-4295-aa4b-7b57cbe7efed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/254ea03c-844c-4295-aa4b-7b57cbe7efed-metrics-certs\") pod \"254ea03c-844c-4295-aa4b-7b57cbe7efed\" (UID: \"254ea03c-844c-4295-aa4b-7b57cbe7efed\") "
Oct 30 01:00:05.121339 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.121229    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/254ea03c-844c-4295-aa4b-7b57cbe7efed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254ea03c-844c-4295-aa4b-7b57cbe7efed-maas-api-tls\") pod \"254ea03c-844c-4295-aa4b-7b57cbe7efed\" (UID: \"254ea03c-844c-4295-aa4b-7b57cbe7efed\") "
Oct 30 01:00:05.121339 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.121284    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-istio-envoy\") pod \"4a920d38-7878-4299-b399-64f3f76d7e55\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") "
Oct 30 01:00:05.121574 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.121345    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-credential-socket\") pod \"4a920d38-7878-4299-b399-64f3f76d7e55\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") "
Oct 30 01:00:05.121574 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.121422    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/254ea03c-844c-4295-aa4b-7b57cbe7efed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254ea03c-844c-4295-aa4b-7b57cbe7efed-openshift-service-ca\") pod \"254ea03c-844c-4295-aa4b-7b57cbe7efed\" (UID: \"254ea03c-844c-4295-aa4b-7b57cbe7efed\") "
Oct 30 01:00:05.121574 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.121475    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a920d38-7878-4299-b399-64f3f76d7e55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a920d38-7878-4299-b399-64f3f76d7e55-istio-podinfo\") pod \"4a920d38-7878-4299-b399-64f3f76d7e55\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") "
Oct 30 01:00:05.121760 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.121586    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a920d38-7878-4299-b399-64f3f76d7e55-kube-api-access-hqmlk\" (UniqueName: \"kubernetes.io/projected/4a920d38-7878-4299-b399-64f3f76d7e55-kube-api-access-hqmlk\") pod \"4a920d38-7878-4299-b399-64f3f76d7e55\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") "
Oct 30 01:00:05.121760 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.121659    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a920d38-7878-4299-b399-64f3f76d7e55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a920d38-7878-4299-b399-64f3f76d7e55-istiod-ca-cert\") pod \"4a920d38-7878-4299-b399-64f3f76d7e55\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") "
Oct 30 01:00:05.121760 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.121737    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-workload-certs\") pod \"4a920d38-7878-4299-b399-64f3f76d7e55\" (UID: \"4a920d38-7878-4299-b399-64f3f76d7e55\") "
Oct 30 01:00:05.124775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.122838    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr9pj\" (UniqueName: \"kubernetes.io/projected/254ea03c-844c-4295-aa4b-7b57cbe7efed-kube-api-access-hr9pj\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.124775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.122868    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.124775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.122885    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.124775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.122898    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.124775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.122912    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.124775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.122927    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.124775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.122940    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.124775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.122953    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e82201b7-f285-4ec3-ba2c-ddb29d770391-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.124775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.122966    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e82201b7-f285-4ec3-ba2c-ddb29d770391-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.124775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.122977    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfgwx\" (UniqueName: \"kubernetes.io/projected/e82201b7-f285-4ec3-ba2c-ddb29d770391-kube-api-access-nfgwx\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.124775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.123634    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a920d38-7878-4299-b399-64f3f76d7e55-istiod-ca-cert" pod "4a920d38-7878-4299-b399-64f3f76d7e55" (UID: "4a920d38-7878-4299-b399-64f3f76d7e55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:05.124775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.124157    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-workload-certs" pod "4a920d38-7878-4299-b399-64f3f76d7e55" (UID: "4a920d38-7878-4299-b399-64f3f76d7e55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:05.127881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.124926    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-istio-data" pod "4a920d38-7878-4299-b399-64f3f76d7e55" (UID: "4a920d38-7878-4299-b399-64f3f76d7e55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:05.127881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.125117    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-credential-socket" pod "4a920d38-7878-4299-b399-64f3f76d7e55" (UID: "4a920d38-7878-4299-b399-64f3f76d7e55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:05.127881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.125132    6099 generic.go:356] "Generic (PLEG): container finished" podID="9c86d85f-10ed-4300-98d7-e6934451d12d" containerID="2deb607e40577b597147bf69eb9084f857e641f88b8b499bb8b6e6be47e8b8db" exitCode=60
Oct 30 01:00:05.127881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.125530    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw" event={"ID":"9c86d85f-10ed-4300-98d7-e6934451d12d","Type":"ContainerDied","Data":"2deb607e40577b597147bf69eb9084f857e641f88b8b499bb8b6e6be47e8b8db"}
Oct 30 01:00:05.127881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.125601    6099 scope.go:122] "RemoveContainer" containerID="3eb15df51a35808a7824a60fb5d144079b209ba84280af8dd9f1a8d6904a9b16"
Oct 30 01:00:05.127881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.125644    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254ea03c-844c-4295-aa4b-7b57cbe7efed-openshift-service-ca" pod "254ea03c-844c-4295-aa4b-7b57cbe7efed" (UID: "254ea03c-844c-4295-aa4b-7b57cbe7efed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:05.127881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.127050    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-workload-socket" pod "4a920d38-7878-4299-b399-64f3f76d7e55" (UID: "4a920d38-7878-4299-b399-64f3f76d7e55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:05.127881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.127644    6099 scope.go:122] "RemoveContainer" containerID="2deb607e40577b597147bf69eb9084f857e641f88b8b499bb8b6e6be47e8b8db"
Oct 30 01:00:05.128586 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:05.128047    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw_odh-ai-gateway-infra(9c86d85f-10ed-4300-98d7-e6934451d12d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw" podUID="9c86d85f-10ed-4300-98d7-e6934451d12d"
Oct 30 01:00:05.129577 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.129551    6099 generic.go:356] "Generic (PLEG): container finished" podID="09a29068-f886-4f57-9eab-db7680a24e7a" containerID="0769681754cb16ec2e57cdc67971934829ee36edf542e457e882443a07a98b94" exitCode=60
Oct 30 01:00:05.129694 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.129653    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7" event={"ID":"09a29068-f886-4f57-9eab-db7680a24e7a","Type":"ContainerDied","Data":"0769681754cb16ec2e57cdc67971934829ee36edf542e457e882443a07a98b94"}
Oct 30 01:00:05.144450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.130718    6099 scope.go:122] "RemoveContainer" containerID="0769681754cb16ec2e57cdc67971934829ee36edf542e457e882443a07a98b94"
Oct 30 01:00:05.144450 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:05.131521    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7_odh-ai-gateway-infra(09a29068-f886-4f57-9eab-db7680a24e7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7" podUID="09a29068-f886-4f57-9eab-db7680a24e7a"
Oct 30 01:00:05.144450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.131680    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254ea03c-844c-4295-aa4b-7b57cbe7efed-maas-api-tls" pod "254ea03c-844c-4295-aa4b-7b57cbe7efed" (UID: "254ea03c-844c-4295-aa4b-7b57cbe7efed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:05.144450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.131754    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a920d38-7878-4299-b399-64f3f76d7e55-istio-token" pod "4a920d38-7878-4299-b399-64f3f76d7e55" (UID: "4a920d38-7878-4299-b399-64f3f76d7e55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:05.144450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.131822    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a920d38-7878-4299-b399-64f3f76d7e55-istio-podinfo" pod "4a920d38-7878-4299-b399-64f3f76d7e55" (UID: "4a920d38-7878-4299-b399-64f3f76d7e55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:00:05.144450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.131903    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254ea03c-844c-4295-aa4b-7b57cbe7efed-metrics-certs" pod "254ea03c-844c-4295-aa4b-7b57cbe7efed" (UID: "254ea03c-844c-4295-aa4b-7b57cbe7efed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:05.144450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.131952    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a920d38-7878-4299-b399-64f3f76d7e55-kube-api-access-hqmlk" pod "4a920d38-7878-4299-b399-64f3f76d7e55" (UID: "4a920d38-7878-4299-b399-64f3f76d7e55"). InnerVolumeSpecName "kube-api-access-hqmlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:05.144450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.132024    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-istio-envoy" pod "4a920d38-7878-4299-b399-64f3f76d7e55" (UID: "4a920d38-7878-4299-b399-64f3f76d7e55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:05.144450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.133394    6099 generic.go:356] "Generic (PLEG): container finished" podID="4a920d38-7878-4299-b399-64f3f76d7e55" containerID="9e0a6e0513f77dadd687fcfba21a0822533c4e25c145b08ae501f2fde03d66d0" exitCode=0
Oct 30 01:00:05.144450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.133548    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px" event={"ID":"4a920d38-7878-4299-b399-64f3f76d7e55","Type":"ContainerDied","Data":"9e0a6e0513f77dadd687fcfba21a0822533c4e25c145b08ae501f2fde03d66d0"}
Oct 30 01:00:05.144450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.133589    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px" event={"ID":"4a920d38-7878-4299-b399-64f3f76d7e55","Type":"ContainerDied","Data":"7b1020bf9d944a58955b325bd336d13ca7eff37e9e0d358b058316696c1ee24b"}
Oct 30 01:00:05.144450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.133669    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"
Oct 30 01:00:05.144450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.140543    6099 generic.go:356] "Generic (PLEG): container finished" podID="e82201b7-f285-4ec3-ba2c-ddb29d770391" containerID="11eb2ba27d3e81ac698a5c7f261a5db3a0f6b163ca92c8f94a03719edb61dba0" exitCode=0
Oct 30 01:00:05.144450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.140629    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj" event={"ID":"e82201b7-f285-4ec3-ba2c-ddb29d770391","Type":"ContainerDied","Data":"11eb2ba27d3e81ac698a5c7f261a5db3a0f6b163ca92c8f94a03719edb61dba0"}
Oct 30 01:00:05.144450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.140682    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj" event={"ID":"e82201b7-f285-4ec3-ba2c-ddb29d770391","Type":"ContainerDied","Data":"4b8e717a71e4f855276c021a87b225a8baa1b1534b3549b871a9ced2b1413045"}
Oct 30 01:00:05.144450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.140814    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"
Oct 30 01:00:05.148433 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.148312    6099 generic.go:356] "Generic (PLEG): container finished" podID="e23eea04-ac89-4fa4-bb29-f38d655df0d7" containerID="d67ec0d59179b350dccaa52c3d47aa626b48b5c1b1d4af70bbcbd1fa8e9fd87d" exitCode=60
Oct 30 01:00:05.148433 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.148351    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k" event={"ID":"e23eea04-ac89-4fa4-bb29-f38d655df0d7","Type":"ContainerDied","Data":"d67ec0d59179b350dccaa52c3d47aa626b48b5c1b1d4af70bbcbd1fa8e9fd87d"}
Oct 30 01:00:05.148782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.148760    6099 scope.go:122] "RemoveContainer" containerID="d67ec0d59179b350dccaa52c3d47aa626b48b5c1b1d4af70bbcbd1fa8e9fd87d"
Oct 30 01:00:05.149437 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:05.149410    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k_odh-ai-gateway-infra(e23eea04-ac89-4fa4-bb29-f38d655df0d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k" podUID="e23eea04-ac89-4fa4-bb29-f38d655df0d7"
Oct 30 01:00:05.178958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.175981    6099 generic.go:356] "Generic (PLEG): container finished" podID="0e2294af-beb7-4d49-8cd8-4abd0c04a40a" containerID="fdfe9116ab85773c9d0fdf56b577a257ad05d0c5df76babeac5f96f334495b9d" exitCode=60
Oct 30 01:00:05.178958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.176162    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl" event={"ID":"0e2294af-beb7-4d49-8cd8-4abd0c04a40a","Type":"ContainerDied","Data":"fdfe9116ab85773c9d0fdf56b577a257ad05d0c5df76babeac5f96f334495b9d"}
Oct 30 01:00:05.178958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.176869    6099 scope.go:122] "RemoveContainer" containerID="fdfe9116ab85773c9d0fdf56b577a257ad05d0c5df76babeac5f96f334495b9d"
Oct 30 01:00:05.178958 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:05.177614    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888700-9p4dl_odh-ai-gateway-infra(0e2294af-beb7-4d49-8cd8-4abd0c04a40a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl" podUID="0e2294af-beb7-4d49-8cd8-4abd0c04a40a"
Oct 30 01:00:05.244197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.239758    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a920d38-7878-4299-b399-64f3f76d7e55-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.244197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.239817    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.244197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.239836    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.244197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.239852    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.244197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.239867    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a920d38-7878-4299-b399-64f3f76d7e55-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.244197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.239880    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254ea03c-844c-4295-aa4b-7b57cbe7efed-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.244197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.239894    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254ea03c-844c-4295-aa4b-7b57cbe7efed-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.244197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.239907    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.244197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.239922    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a920d38-7878-4299-b399-64f3f76d7e55-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.244197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.239938    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254ea03c-844c-4295-aa4b-7b57cbe7efed-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.244197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.239952    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a920d38-7878-4299-b399-64f3f76d7e55-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.244197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.239967    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqmlk\" (UniqueName: \"kubernetes.io/projected/4a920d38-7878-4299-b399-64f3f76d7e55-kube-api-access-hqmlk\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:05.270385 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.270346    6099 generic.go:356] "Generic (PLEG): container finished" podID="254ea03c-844c-4295-aa4b-7b57cbe7efed" containerID="3873b8b3b12c2553fa42e2827ace0123907174bb839eea5aa3abe71580f14304" exitCode=0
Oct 30 01:00:05.270926 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.270881    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"
Oct 30 01:00:05.271198 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.271175    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps" event={"ID":"254ea03c-844c-4295-aa4b-7b57cbe7efed","Type":"ContainerDied","Data":"3873b8b3b12c2553fa42e2827ace0123907174bb839eea5aa3abe71580f14304"}
Oct 30 01:00:05.271268 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.271233    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps" event={"ID":"254ea03c-844c-4295-aa4b-7b57cbe7efed","Type":"ContainerDied","Data":"418d5cff240fc3652af63be38b1c77cd18065c388b34e5b2312aafc20e156e02"}
Oct 30 01:00:05.273877 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.273566    6099 scope.go:122] "RemoveContainer" containerID="31bb6b5d454f4852c3298c03a9a564a2e8406066486608ea63d1cc1df7b16d00"
Oct 30 01:00:05.303036 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.302795    6099 scope.go:122] "RemoveContainer" containerID="bd3362ec4618f45cf825b644f1f34d409f52471a163ddc3a72c47513daa47d86"
Oct 30 01:00:05.339207 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.339112    6099 scope.go:122] "RemoveContainer" containerID="fabe165d9cb30a76572df1442604e4eb72759d6f4d1d1e873571549887cb2bd7"
Oct 30 01:00:05.363585 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.363560    6099 scope.go:122] "RemoveContainer" containerID="f47ea9728c54cf2e278ef078128a700754551fb12ddeb26aca039426672aeed1"
Oct 30 01:00:05.384489 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.384349    6099 scope.go:122] "RemoveContainer" containerID="9e0a6e0513f77dadd687fcfba21a0822533c4e25c145b08ae501f2fde03d66d0"
Oct 30 01:00:05.428207 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.427842    6099 scope.go:122] "RemoveContainer" containerID="89e1b21db3e8f12e1d52d09db4842f82174974bd6fd0c11bbd84bcffe7a11d75"
Oct 30 01:00:05.450534 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.450468    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"]
Oct 30 01:00:05.465899 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.463755    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf"]
Oct 30 01:00:05.477423 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.477355    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73b32e-67b8b984f5-5txps"]
Oct 30 01:00:05.483573 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.483543    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-73b32e-n7zlf"]
Oct 30 01:00:05.487471 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.487445    6099 scope.go:122] "RemoveContainer" containerID="9e0a6e0513f77dadd687fcfba21a0822533c4e25c145b08ae501f2fde03d66d0"
Oct 30 01:00:05.488027 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:05.487896    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0a6e0513f77dadd687fcfba21a0822533c4e25c145b08ae501f2fde03d66d0\": container with ID starting with 9e0a6e0513f77dadd687fcfba21a0822533c4e25c145b08ae501f2fde03d66d0 not found: ID does not exist" containerID="9e0a6e0513f77dadd687fcfba21a0822533c4e25c145b08ae501f2fde03d66d0"
Oct 30 01:00:05.488637 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.488589    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0a6e0513f77dadd687fcfba21a0822533c4e25c145b08ae501f2fde03d66d0"} err="failed to get container status \"9e0a6e0513f77dadd687fcfba21a0822533c4e25c145b08ae501f2fde03d66d0\": rpc error: code = NotFound desc = could not find container \"9e0a6e0513f77dadd687fcfba21a0822533c4e25c145b08ae501f2fde03d66d0\": container with ID starting with 9e0a6e0513f77dadd687fcfba21a0822533c4e25c145b08ae501f2fde03d66d0 not found: ID does not exist"
Oct 30 01:00:05.488787 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.488641    6099 scope.go:122] "RemoveContainer" containerID="89e1b21db3e8f12e1d52d09db4842f82174974bd6fd0c11bbd84bcffe7a11d75"
Oct 30 01:00:05.489160 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:05.489103    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e1b21db3e8f12e1d52d09db4842f82174974bd6fd0c11bbd84bcffe7a11d75\": container with ID starting with 89e1b21db3e8f12e1d52d09db4842f82174974bd6fd0c11bbd84bcffe7a11d75 not found: ID does not exist" containerID="89e1b21db3e8f12e1d52d09db4842f82174974bd6fd0c11bbd84bcffe7a11d75"
Oct 30 01:00:05.489160 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.489143    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e1b21db3e8f12e1d52d09db4842f82174974bd6fd0c11bbd84bcffe7a11d75"} err="failed to get container status \"89e1b21db3e8f12e1d52d09db4842f82174974bd6fd0c11bbd84bcffe7a11d75\": rpc error: code = NotFound desc = could not find container \"89e1b21db3e8f12e1d52d09db4842f82174974bd6fd0c11bbd84bcffe7a11d75\": container with ID starting with 89e1b21db3e8f12e1d52d09db4842f82174974bd6fd0c11bbd84bcffe7a11d75 not found: ID does not exist"
Oct 30 01:00:05.489357 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.489166    6099 scope.go:122] "RemoveContainer" containerID="11eb2ba27d3e81ac698a5c7f261a5db3a0f6b163ca92c8f94a03719edb61dba0"
Oct 30 01:00:05.511765 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.511357    6099 scope.go:122] "RemoveContainer" containerID="11eb2ba27d3e81ac698a5c7f261a5db3a0f6b163ca92c8f94a03719edb61dba0"
Oct 30 01:00:05.511905 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:05.511770    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11eb2ba27d3e81ac698a5c7f261a5db3a0f6b163ca92c8f94a03719edb61dba0\": container with ID starting with 11eb2ba27d3e81ac698a5c7f261a5db3a0f6b163ca92c8f94a03719edb61dba0 not found: ID does not exist" containerID="11eb2ba27d3e81ac698a5c7f261a5db3a0f6b163ca92c8f94a03719edb61dba0"
Oct 30 01:00:05.511905 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.511804    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11eb2ba27d3e81ac698a5c7f261a5db3a0f6b163ca92c8f94a03719edb61dba0"} err="failed to get container status \"11eb2ba27d3e81ac698a5c7f261a5db3a0f6b163ca92c8f94a03719edb61dba0\": rpc error: code = NotFound desc = could not find container \"11eb2ba27d3e81ac698a5c7f261a5db3a0f6b163ca92c8f94a03719edb61dba0\": container with ID starting with 11eb2ba27d3e81ac698a5c7f261a5db3a0f6b163ca92c8f94a03719edb61dba0 not found: ID does not exist"
Oct 30 01:00:05.511905 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.511829    6099 scope.go:122] "RemoveContainer" containerID="80732e41ca073ff38f177eb1806ebabdb81a9ff6737b81104d0922aee9cd70ec"
Oct 30 01:00:05.531124 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.531090    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"]
Oct 30 01:00:05.531942 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.531921    6099 scope.go:122] "RemoveContainer" containerID="6a271fc2164e1dadaf2a53dfa3823ba0790e1554ef64adb5b1f3adb6ed0442a1"
Oct 30 01:00:05.536923 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.536886    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-q56px"]
Oct 30 01:00:05.547219 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.547203    6099 scope.go:122] "RemoveContainer" containerID="3873b8b3b12c2553fa42e2827ace0123907174bb839eea5aa3abe71580f14304"
Oct 30 01:00:05.569040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.569009    6099 scope.go:122] "RemoveContainer" containerID="3873b8b3b12c2553fa42e2827ace0123907174bb839eea5aa3abe71580f14304"
Oct 30 01:00:05.569385 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:05.569363    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3873b8b3b12c2553fa42e2827ace0123907174bb839eea5aa3abe71580f14304\": container with ID starting with 3873b8b3b12c2553fa42e2827ace0123907174bb839eea5aa3abe71580f14304 not found: ID does not exist" containerID="3873b8b3b12c2553fa42e2827ace0123907174bb839eea5aa3abe71580f14304"
Oct 30 01:00:05.569433 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.569396    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3873b8b3b12c2553fa42e2827ace0123907174bb839eea5aa3abe71580f14304"} err="failed to get container status \"3873b8b3b12c2553fa42e2827ace0123907174bb839eea5aa3abe71580f14304\": rpc error: code = NotFound desc = could not find container \"3873b8b3b12c2553fa42e2827ace0123907174bb839eea5aa3abe71580f14304\": container with ID starting with 3873b8b3b12c2553fa42e2827ace0123907174bb839eea5aa3abe71580f14304 not found: ID does not exist"
Oct 30 01:00:05.651681 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.651643    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"]
Oct 30 01:00:05.657094 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:05.657063    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-353403-openshift-default-5f567587b9-sfvmj"]
Oct 30 01:00:06.266609 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:06.266578    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="254ea03c-844c-4295-aa4b-7b57cbe7efed" path="/var/lib/kubelet/pods/254ea03c-844c-4295-aa4b-7b57cbe7efed/volumes"
Oct 30 01:00:06.267302 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:06.267280    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d90b85-8170-47ab-bcdf-7d897eba3e7f" path="/var/lib/kubelet/pods/46d90b85-8170-47ab-bcdf-7d897eba3e7f/volumes"
Oct 30 01:00:06.267847 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:06.267823    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a920d38-7878-4299-b399-64f3f76d7e55" path="/var/lib/kubelet/pods/4a920d38-7878-4299-b399-64f3f76d7e55/volumes"
Oct 30 01:00:06.268658 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:06.268637    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e82201b7-f285-4ec3-ba2c-ddb29d770391" path="/var/lib/kubelet/pods/e82201b7-f285-4ec3-ba2c-ddb29d770391/volumes"
Oct 30 01:00:06.280230 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:06.280208    6099 scope.go:122] "RemoveContainer" containerID="fdfe9116ab85773c9d0fdf56b577a257ad05d0c5df76babeac5f96f334495b9d"
Oct 30 01:00:06.280451 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:06.280430    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888700-9p4dl_odh-ai-gateway-infra(0e2294af-beb7-4d49-8cd8-4abd0c04a40a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl" podUID="0e2294af-beb7-4d49-8cd8-4abd0c04a40a"
Oct 30 01:00:06.282686 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:06.282667    6099 scope.go:122] "RemoveContainer" containerID="2f86196dc2acb9a1ca5a79d3cd75b1b22ee6257de4798d53c28e593e6855b96a"
Oct 30 01:00:06.282955 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:06.282932    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl_odh-ai-gateway-infra(3365bf0b-a96c-429e-8565-faa44fef8ee0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl" podUID="3365bf0b-a96c-429e-8565-faa44fef8ee0"
Oct 30 01:00:06.284366 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:06.284347    6099 scope.go:122] "RemoveContainer" containerID="4a1dd6cb9cf50a6d1c5442604282d5504e508652d61ad1f2a648c2ed93add066"
Oct 30 01:00:06.284587 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:06.284566    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc_odh-ai-gateway-infra(ed2d10ae-6748-461e-92c4-f5efc2f0c880)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc" podUID="ed2d10ae-6748-461e-92c4-f5efc2f0c880"
Oct 30 01:00:06.285960 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:06.285944    6099 scope.go:122] "RemoveContainer" containerID="97bdba38dd333e0e077dc6bc4ddbad1d9eaae9e898fc56e46b79b4ecafa9c486"
Oct 30 01:00:06.286185 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:06.286161    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz_odh-ai-gateway-infra(f378df0c-bde8-4326-ade3-7c4ecd956cb3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz" podUID="f378df0c-bde8-4326-ade3-7c4ecd956cb3"
Oct 30 01:00:06.287591 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:06.287574    6099 scope.go:122] "RemoveContainer" containerID="f20075a6f5d98bfae69954722ec529e46eb1a8d32b4b163ce4780e63dc036d85"
Oct 30 01:00:06.287855 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:06.287831    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2_odh-ai-gateway-infra(fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2" podUID="fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9"
Oct 30 01:00:06.289212 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:06.289196    6099 scope.go:122] "RemoveContainer" containerID="2deb607e40577b597147bf69eb9084f857e641f88b8b499bb8b6e6be47e8b8db"
Oct 30 01:00:06.289429 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:06.289405    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw_odh-ai-gateway-infra(9c86d85f-10ed-4300-98d7-e6934451d12d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw" podUID="9c86d85f-10ed-4300-98d7-e6934451d12d"
Oct 30 01:00:06.290779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:06.290760    6099 scope.go:122] "RemoveContainer" containerID="0769681754cb16ec2e57cdc67971934829ee36edf542e457e882443a07a98b94"
Oct 30 01:00:06.291021 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:06.290999    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7_odh-ai-gateway-infra(09a29068-f886-4f57-9eab-db7680a24e7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7" podUID="09a29068-f886-4f57-9eab-db7680a24e7a"
Oct 30 01:00:06.294280 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:06.294253    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw" podUID="020ef144-858f-4125-a9b8-f18fbebbffad" containerName="cleanup" containerID="cri-o://a331f8481dc3d340991fa433417ca5d071e346625bd5e76932ede083f43e4da9" gracePeriod=30
Oct 30 01:00:06.294396 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:06.294282    6099 scope.go:122] "RemoveContainer" containerID="d67ec0d59179b350dccaa52c3d47aa626b48b5c1b1d4af70bbcbd1fa8e9fd87d"
Oct 30 01:00:06.294894 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:06.294527    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k_odh-ai-gateway-infra(e23eea04-ac89-4fa4-bb29-f38d655df0d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k" podUID="e23eea04-ac89-4fa4-bb29-f38d655df0d7"
Oct 30 01:00:06.544948 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:06.544921    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 01:00:06.556558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:06.556534    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 01:00:07.551552 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.551520    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"]
Oct 30 01:00:07.552317 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.552301    6099 cpu_manager.go:414] "containerMap: removing container" podUID="4a920d38-7878-4299-b399-64f3f76d7e55" containerName="istio-proxy"
Oct 30 01:00:07.552362 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.552318    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a920d38-7878-4299-b399-64f3f76d7e55" containerName="istio-proxy"
Oct 30 01:00:07.552362 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.552336    6099 cpu_manager.go:414] "containerMap: removing container" podUID="254ea03c-844c-4295-aa4b-7b57cbe7efed" containerName="maas-api"
Oct 30 01:00:07.552362 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.552342    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254ea03c-844c-4295-aa4b-7b57cbe7efed" containerName="maas-api"
Oct 30 01:00:07.552362 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.552356    6099 cpu_manager.go:414] "containerMap: removing container" podUID="4a920d38-7878-4299-b399-64f3f76d7e55" containerName="istio-proxy"
Oct 30 01:00:07.552489 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.552364    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a920d38-7878-4299-b399-64f3f76d7e55" containerName="istio-proxy"
Oct 30 01:00:07.552489 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.552377    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e82201b7-f285-4ec3-ba2c-ddb29d770391" containerName="istio-proxy"
Oct 30 01:00:07.552489 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.552381    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e82201b7-f285-4ec3-ba2c-ddb29d770391" containerName="istio-proxy"
Oct 30 01:00:07.552489 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.552390    6099 cpu_manager.go:414] "containerMap: removing container" podUID="6370d09b-ecc5-4ca5-968f-d39f3bdefcde" containerName="revoke-keys"
Oct 30 01:00:07.552489 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.552394    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6370d09b-ecc5-4ca5-968f-d39f3bdefcde" containerName="revoke-keys"
Oct 30 01:00:07.552640 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.552508    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a920d38-7878-4299-b399-64f3f76d7e55" containerName="istio-proxy"
Oct 30 01:00:07.552640 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.552518    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="254ea03c-844c-4295-aa4b-7b57cbe7efed" containerName="maas-api"
Oct 30 01:00:07.552640 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.552528    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a920d38-7878-4299-b399-64f3f76d7e55" containerName="istio-proxy"
Oct 30 01:00:07.552640 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.552534    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e82201b7-f285-4ec3-ba2c-ddb29d770391" containerName="istio-proxy"
Oct 30 01:00:07.552640 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.552541    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6370d09b-ecc5-4ca5-968f-d39f3bdefcde" containerName="revoke-keys"
Oct 30 01:00:07.558121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.558105    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"
Oct 30 01:00:07.578278 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.578251    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"]
Oct 30 01:00:07.585778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.585746    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-43c02d-serving-cert\""
Oct 30 01:00:07.694376 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.694331    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49f719fe-16aa-431d-91b7-1730267521ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx\" (UID: \"49f719fe-16aa-431d-91b7-1730267521ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"
Oct 30 01:00:07.694566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.694452    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmc4m\" (UniqueName: \"kubernetes.io/projected/49f719fe-16aa-431d-91b7-1730267521ec-kube-api-access-dmc4m\") pod \"maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx\" (UID: \"49f719fe-16aa-431d-91b7-1730267521ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"
Oct 30 01:00:07.694687 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.694667    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f719fe-16aa-431d-91b7-1730267521ec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx\" (UID: \"49f719fe-16aa-431d-91b7-1730267521ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"
Oct 30 01:00:07.694823 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.694808    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49f719fe-16aa-431d-91b7-1730267521ec-metrics-certs\") pod \"maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx\" (UID: \"49f719fe-16aa-431d-91b7-1730267521ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"
Oct 30 01:00:07.799173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.797003    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49f719fe-16aa-431d-91b7-1730267521ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx\" (UID: \"49f719fe-16aa-431d-91b7-1730267521ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"
Oct 30 01:00:07.799173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.797090    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmc4m\" (UniqueName: \"kubernetes.io/projected/49f719fe-16aa-431d-91b7-1730267521ec-kube-api-access-dmc4m\") pod \"maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx\" (UID: \"49f719fe-16aa-431d-91b7-1730267521ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"
Oct 30 01:00:07.799173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.797313    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f719fe-16aa-431d-91b7-1730267521ec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx\" (UID: \"49f719fe-16aa-431d-91b7-1730267521ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"
Oct 30 01:00:07.799173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.797409    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49f719fe-16aa-431d-91b7-1730267521ec-metrics-certs\") pod \"maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx\" (UID: \"49f719fe-16aa-431d-91b7-1730267521ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"
Oct 30 01:00:07.799173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.798260    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f719fe-16aa-431d-91b7-1730267521ec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx\" (UID: \"49f719fe-16aa-431d-91b7-1730267521ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"
Oct 30 01:00:07.815575 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.815486    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49f719fe-16aa-431d-91b7-1730267521ec-metrics-certs\") pod \"maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx\" (UID: \"49f719fe-16aa-431d-91b7-1730267521ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"
Oct 30 01:00:07.816849 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.816808    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49f719fe-16aa-431d-91b7-1730267521ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx\" (UID: \"49f719fe-16aa-431d-91b7-1730267521ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"
Oct 30 01:00:07.821770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.821686    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmc4m\" (UniqueName: \"kubernetes.io/projected/49f719fe-16aa-431d-91b7-1730267521ec-kube-api-access-dmc4m\") pod \"maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx\" (UID: \"49f719fe-16aa-431d-91b7-1730267521ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"
Oct 30 01:00:07.830128 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.830089    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg" podUID="4f260864-36c4-4a59-b3a9-b47332908825" containerName="payload-processing" containerID="cri-o://1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af" gracePeriod=25
Oct 30 01:00:07.864258 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.864223    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq" podUID="4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a" containerName="payload-pre-processing" containerID="cri-o://3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5" gracePeriod=25
Oct 30 01:00:07.871981 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:07.871952    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"
Oct 30 01:00:08.193772 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.193742    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"]
Oct 30 01:00:08.215017 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.214990    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg"
Oct 30 01:00:08.220326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.220305    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq"
Oct 30 01:00:08.306736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.306416    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a-plugins-config-volume\") pod \"4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a\" (UID: \"4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a\") "
Oct 30 01:00:08.306736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.306483    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f260864-36c4-4a59-b3a9-b47332908825-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f260864-36c4-4a59-b3a9-b47332908825-plugins-config-volume\") pod \"4f260864-36c4-4a59-b3a9-b47332908825\" (UID: \"4f260864-36c4-4a59-b3a9-b47332908825\") "
Oct 30 01:00:08.308393 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.307092    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a-kube-api-access-4zlbj\" (UniqueName: \"kubernetes.io/projected/4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a-kube-api-access-4zlbj\") pod \"4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a\" (UID: \"4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a\") "
Oct 30 01:00:08.308393 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.307146    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f260864-36c4-4a59-b3a9-b47332908825-kube-api-access-z7qj2\" (UniqueName: \"kubernetes.io/projected/4f260864-36c4-4a59-b3a9-b47332908825-kube-api-access-z7qj2\") pod \"4f260864-36c4-4a59-b3a9-b47332908825\" (UID: \"4f260864-36c4-4a59-b3a9-b47332908825\") "
Oct 30 01:00:08.309043 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.309010    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a-plugins-config-volume" pod "4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a" (UID: "4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:08.311473 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.311452    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f260864-36c4-4a59-b3a9-b47332908825-plugins-config-volume" pod "4f260864-36c4-4a59-b3a9-b47332908825" (UID: "4f260864-36c4-4a59-b3a9-b47332908825"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:08.327851 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.327802    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a-kube-api-access-4zlbj" pod "4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a" (UID: "4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a"). InnerVolumeSpecName "kube-api-access-4zlbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:08.328081 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.328057    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f260864-36c4-4a59-b3a9-b47332908825-kube-api-access-z7qj2" pod "4f260864-36c4-4a59-b3a9-b47332908825" (UID: "4f260864-36c4-4a59-b3a9-b47332908825"). InnerVolumeSpecName "kube-api-access-z7qj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:08.333070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.333038    6099 generic.go:356] "Generic (PLEG): container finished" podID="4f260864-36c4-4a59-b3a9-b47332908825" containerID="1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af" exitCode=0
Oct 30 01:00:08.333193 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.333078    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg"
Oct 30 01:00:08.333193 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.333121    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg" event={"ID":"4f260864-36c4-4a59-b3a9-b47332908825","Type":"ContainerDied","Data":"1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af"}
Oct 30 01:00:08.333193 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.333174    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg" event={"ID":"4f260864-36c4-4a59-b3a9-b47332908825","Type":"ContainerDied","Data":"282b21726f130a621a472ee44fd75b56172105a5a97847b7a5cb1803bcbfc181"}
Oct 30 01:00:08.333340 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.333201    6099 scope.go:122] "RemoveContainer" containerID="1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af"
Oct 30 01:00:08.335536 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.335511    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx" event={"ID":"49f719fe-16aa-431d-91b7-1730267521ec","Type":"ContainerStarted","Data":"aa2e14f38fb3be577a959e643daa37061c26445883d48054fb9c28f61b8b05d1"}
Oct 30 01:00:08.340156 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.340122    6099 generic.go:356] "Generic (PLEG): container finished" podID="4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a" containerID="3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5" exitCode=0
Oct 30 01:00:08.340261 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.340183    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq"
Oct 30 01:00:08.340261 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.340233    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq" event={"ID":"4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a","Type":"ContainerDied","Data":"3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5"}
Oct 30 01:00:08.340359 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.340274    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq" event={"ID":"4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a","Type":"ContainerDied","Data":"7e2678fa240ef4837699a8f6bae1f5cfbaa1664f174092e6001d8be60e30580b"}
Oct 30 01:00:08.350220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.350193    6099 scope.go:122] "RemoveContainer" containerID="1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af"
Oct 30 01:00:08.350530 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:08.350507    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af\": container with ID starting with 1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af not found: ID does not exist" containerID="1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af"
Oct 30 01:00:08.350605 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.350540    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af"} err="failed to get container status \"1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af\": rpc error: code = NotFound desc = could not find container \"1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af\": container with ID starting with 1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af not found: ID does not exist"
Oct 30 01:00:08.350605 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.350569    6099 scope.go:122] "RemoveContainer" containerID="3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5"
Oct 30 01:00:08.394996 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.394976    6099 scope.go:122] "RemoveContainer" containerID="3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5"
Oct 30 01:00:08.395287 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:08.395268    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5\": container with ID starting with 3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5 not found: ID does not exist" containerID="3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5"
Oct 30 01:00:08.395359 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.395293    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5"} err="failed to get container status \"3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5\": rpc error: code = NotFound desc = could not find container \"3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5\": container with ID starting with 3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5 not found: ID does not exist"
Oct 30 01:00:08.408431 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.408399    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg"]
Oct 30 01:00:08.409534 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.409511    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zlbj\" (UniqueName: \"kubernetes.io/projected/4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a-kube-api-access-4zlbj\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:08.409534 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.409533    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7qj2\" (UniqueName: \"kubernetes.io/projected/4f260864-36c4-4a59-b3a9-b47332908825-kube-api-access-z7qj2\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:08.409728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.409544    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:08.409728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.409555    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f260864-36c4-4a59-b3a9-b47332908825-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:08.413064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.413040    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg"]
Oct 30 01:00:08.452030 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.452006    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq"]
Oct 30 01:00:08.458464 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:08.458441    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq"]
Oct 30 01:00:09.355220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:09.355180    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx" event={"ID":"49f719fe-16aa-431d-91b7-1730267521ec","Type":"ContainerStarted","Data":"777fabb188f7261f434fae363c14751b7e0e5b89686ce4fbd12fedddb59ee9eb"}
Oct 30 01:00:09.355741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:09.355263    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"
Oct 30 01:00:09.399428 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:09.399379    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx" podStartSLOduration=2.091292623 podStartE2EDuration="2.399364675s" totalImagesPullingTime="308.072052ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:09.395888922 +0000 UTC m=+1503.252796682" watchObservedRunningTime="2026-10-30 01:00:09.399364675 +0000 UTC m=+1503.256272442"
Oct 30 01:00:10.180022 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:10.179976    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m"]
Oct 30 01:00:10.213413 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:10.213379    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4"]
Oct 30 01:00:10.266290 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:10.266255    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f260864-36c4-4a59-b3a9-b47332908825" path="/var/lib/kubelet/pods/4f260864-36c4-4a59-b3a9-b47332908825/volumes"
Oct 30 01:00:10.266928 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:10.266901    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a" path="/var/lib/kubelet/pods/4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a/volumes"
Oct 30 01:00:10.555311 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:10.555280    6099 scope.go:122] "RemoveContainer" containerID="0d1258dbc93773aafde5a9e5e4c3736c4944d44a98121352bf1d7feb62951d87"
Oct 30 01:00:10.571307 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:10.571277    6099 scope.go:122] "RemoveContainer" containerID="88322e852822ec19933bbe4224ac3deccfbcf63716205979d39d93811bea801e"
Oct 30 01:00:10.582911 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:10.582889    6099 scope.go:122] "RemoveContainer" containerID="4879c730b4703774380426dc34cbea7614bcbd394c872deba21fbad00afbcd41"
Oct 30 01:00:10.593392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:10.593365    6099 scope.go:122] "RemoveContainer" containerID="0a4e00bba51101601851cdfd2d40c26b5bc765856313ba16e3d43f2009c6715b"
Oct 30 01:00:12.682581 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:12.682549    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"]
Oct 30 01:00:12.683105 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:12.682799    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw" podUID="19e9d22c-b2b4-4795-be11-ff2d98383a92" containerName="istio-proxy" containerID="cri-o://8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853" gracePeriod=30
Oct 30 01:00:12.696522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:12.696493    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"]
Oct 30 01:00:12.696766 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:12.696736    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj" podUID="6474c172-c1ba-43d1-a965-242e6b93a3a2" containerName="istio-proxy" containerID="cri-o://59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764" gracePeriod=30
Oct 30 01:00:13.978696 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:13.976379    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"]
Oct 30 01:00:13.978696 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:13.978460    6099 cpu_manager.go:414] "containerMap: removing container" podUID="4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a" containerName="payload-pre-processing"
Oct 30 01:00:13.978696 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:13.978483    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a" containerName="payload-pre-processing"
Oct 30 01:00:13.978696 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:13.978595    6099 cpu_manager.go:414] "containerMap: removing container" podUID="4f260864-36c4-4a59-b3a9-b47332908825" containerName="payload-processing"
Oct 30 01:00:13.978696 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:13.978605    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f260864-36c4-4a59-b3a9-b47332908825" containerName="payload-processing"
Oct 30 01:00:13.979403 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:13.979033    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f260864-36c4-4a59-b3a9-b47332908825" containerName="payload-processing"
Oct 30 01:00:13.979403 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:13.979072    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a" containerName="payload-pre-processing"
Oct 30 01:00:13.990341 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:13.990318    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.002738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.001717    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"]
Oct 30 01:00:14.008536 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.008510    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-780077-openshift-default-dockercfg-5jlcn\""
Oct 30 01:00:14.131771 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.124843    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbhhh\" (UniqueName: \"kubernetes.io/projected/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-kube-api-access-vbhhh\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.132215 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.132177    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-podinfo\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.132439 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.132418    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istiod-ca-cert\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.132613 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.132596    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-credential-socket\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.132775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.132756    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-token\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.133017 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.132994    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-workload-socket\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.133419 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.133395    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-workload-certs\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.133640 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.133620    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-data\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.136134 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.136106    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-envoy\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.238028 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.237926    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-data\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.238195 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.238071    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-envoy\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.238195 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.238164    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhhh\" (UniqueName: \"kubernetes.io/projected/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-kube-api-access-vbhhh\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.238368 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.238256    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-podinfo\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.238368 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.238300    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istiod-ca-cert\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.238451 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.238399    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-data\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.238535 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.238503    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-credential-socket\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.238599 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.238584    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-token\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.238662 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.238646    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-workload-socket\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.238794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.238777    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-workload-certs\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.238899 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.238876    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-credential-socket\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.239182 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.239155    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-workload-certs\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.239401 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.239363    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-workload-socket\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.239599 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.239385    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istiod-ca-cert\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.241547 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.241513    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-envoy\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.241761 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.241695    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-podinfo\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.264551 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.264522    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-token\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.264812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.264757    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbhhh\" (UniqueName: \"kubernetes.io/projected/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-kube-api-access-vbhhh\") pod \"e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") " pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.325634 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.325590    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:14.658827 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.658801    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"]
Oct 30 01:00:14.661073 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.661041    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:14.661151 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.661117    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:14.661191 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:14.661158    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:15.180821 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.180742    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m" podUID="20857824-d001-4494-97f3-ff58b868c3c2" containerName="payload-processing" containerID="cri-o://21b2b68e62de806c328d1946643254519cab3e76f988257b0e72908f9ab25477" gracePeriod=25
Oct 30 01:00:15.214390 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.214347    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4" podUID="0e190a98-17fe-46f6-adcf-34223eaf820e" containerName="payload-pre-processing" containerID="cri-o://28a44dc7433d5dd5158e619d7485433c6e041166705909aea99f6cb0bec6ea48" gracePeriod=25
Oct 30 01:00:15.372792 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.372762    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"
Oct 30 01:00:15.417495 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.417291    6099 generic.go:356] "Generic (PLEG): container finished" podID="20857824-d001-4494-97f3-ff58b868c3c2" containerID="21b2b68e62de806c328d1946643254519cab3e76f988257b0e72908f9ab25477" exitCode=0
Oct 30 01:00:15.417495 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.417393    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m" event={"ID":"20857824-d001-4494-97f3-ff58b868c3c2","Type":"ContainerDied","Data":"21b2b68e62de806c328d1946643254519cab3e76f988257b0e72908f9ab25477"}
Oct 30 01:00:15.421129 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.421099    6099 generic.go:356] "Generic (PLEG): container finished" podID="0e190a98-17fe-46f6-adcf-34223eaf820e" containerID="28a44dc7433d5dd5158e619d7485433c6e041166705909aea99f6cb0bec6ea48" exitCode=0
Oct 30 01:00:15.421358 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.421296    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4" event={"ID":"0e190a98-17fe-46f6-adcf-34223eaf820e","Type":"ContainerDied","Data":"28a44dc7433d5dd5158e619d7485433c6e041166705909aea99f6cb0bec6ea48"}
Oct 30 01:00:15.425167 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.424830    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm" event={"ID":"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd","Type":"ContainerStarted","Data":"8999a5abb33e86d850a8a12f4f903b3d49b2ee8ed22cdb3834419a048758b9fd"}
Oct 30 01:00:15.425167 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.424882    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm" event={"ID":"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd","Type":"ContainerStarted","Data":"ccb1d5005ffc34e1571c2848fe0eb3f0708ed3fe1d50a4b75349373a40ba0faf"}
Oct 30 01:00:15.477283 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.477239    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm" podStartSLOduration=2.47722623 podStartE2EDuration="2.47722623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:15.471251699 +0000 UTC m=+1509.328159455" watchObservedRunningTime="2026-10-30 01:00:15.47722623 +0000 UTC m=+1509.334134028"
Oct 30 01:00:15.514386 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.514359    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m"
Oct 30 01:00:15.560510 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.560479    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20857824-d001-4494-97f3-ff58b868c3c2-kube-api-access-kfxhg\" (UniqueName: \"kubernetes.io/projected/20857824-d001-4494-97f3-ff58b868c3c2-kube-api-access-kfxhg\") pod \"20857824-d001-4494-97f3-ff58b868c3c2\" (UID: \"20857824-d001-4494-97f3-ff58b868c3c2\") "
Oct 30 01:00:15.560662 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.560602    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20857824-d001-4494-97f3-ff58b868c3c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20857824-d001-4494-97f3-ff58b868c3c2-plugins-config-volume\") pod \"20857824-d001-4494-97f3-ff58b868c3c2\" (UID: \"20857824-d001-4494-97f3-ff58b868c3c2\") "
Oct 30 01:00:15.561022 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.560997    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20857824-d001-4494-97f3-ff58b868c3c2-plugins-config-volume" pod "20857824-d001-4494-97f3-ff58b868c3c2" (UID: "20857824-d001-4494-97f3-ff58b868c3c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:15.561405 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.561387    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20857824-d001-4494-97f3-ff58b868c3c2-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:15.562968 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.562940    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20857824-d001-4494-97f3-ff58b868c3c2-kube-api-access-kfxhg" pod "20857824-d001-4494-97f3-ff58b868c3c2" (UID: "20857824-d001-4494-97f3-ff58b868c3c2"). InnerVolumeSpecName "kube-api-access-kfxhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:15.594815 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.594792    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4"
Oct 30 01:00:15.651820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.651554    6099 patch_prober.go:29] interesting pod/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:00:15.651820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.651608    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw" podUID="19e9d22c-b2b4-4795-be11-ff2d98383a92" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:00:15.662859 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.662826    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e190a98-17fe-46f6-adcf-34223eaf820e-kube-api-access-h45xr\" (UniqueName: \"kubernetes.io/projected/0e190a98-17fe-46f6-adcf-34223eaf820e-kube-api-access-h45xr\") pod \"0e190a98-17fe-46f6-adcf-34223eaf820e\" (UID: \"0e190a98-17fe-46f6-adcf-34223eaf820e\") "
Oct 30 01:00:15.663233 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.663205    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e190a98-17fe-46f6-adcf-34223eaf820e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e190a98-17fe-46f6-adcf-34223eaf820e-plugins-config-volume\") pod \"0e190a98-17fe-46f6-adcf-34223eaf820e\" (UID: \"0e190a98-17fe-46f6-adcf-34223eaf820e\") "
Oct 30 01:00:15.663662 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.663633    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e190a98-17fe-46f6-adcf-34223eaf820e-plugins-config-volume" pod "0e190a98-17fe-46f6-adcf-34223eaf820e" (UID: "0e190a98-17fe-46f6-adcf-34223eaf820e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:15.664140 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.664114    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e190a98-17fe-46f6-adcf-34223eaf820e-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:15.664240 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.664145    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfxhg\" (UniqueName: \"kubernetes.io/projected/20857824-d001-4494-97f3-ff58b868c3c2-kube-api-access-kfxhg\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:15.665216 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.665186    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e190a98-17fe-46f6-adcf-34223eaf820e-kube-api-access-h45xr" pod "0e190a98-17fe-46f6-adcf-34223eaf820e" (UID: "0e190a98-17fe-46f6-adcf-34223eaf820e"). InnerVolumeSpecName "kube-api-access-h45xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:15.765940 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:15.765849    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h45xr\" (UniqueName: \"kubernetes.io/projected/0e190a98-17fe-46f6-adcf-34223eaf820e-kube-api-access-h45xr\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:16.266860 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:16.266830    6099 scope.go:122] "RemoveContainer" containerID="d67ec0d59179b350dccaa52c3d47aa626b48b5c1b1d4af70bbcbd1fa8e9fd87d"
Oct 30 01:00:16.267336 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:16.267089    6099 scope.go:122] "RemoveContainer" containerID="2f86196dc2acb9a1ca5a79d3cd75b1b22ee6257de4798d53c28e593e6855b96a"
Oct 30 01:00:16.267541 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:16.267437    6099 scope.go:122] "RemoveContainer" containerID="fdfe9116ab85773c9d0fdf56b577a257ad05d0c5df76babeac5f96f334495b9d"
Oct 30 01:00:16.326097 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:16.326073    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:16.332089 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:16.332069    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:16.435078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:16.435043    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m"
Oct 30 01:00:16.435249 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:16.435050    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m" event={"ID":"20857824-d001-4494-97f3-ff58b868c3c2","Type":"ContainerDied","Data":"6ff55f112fc5499244152c6694cf55212cc4844ab1b85f0cb3b8fe3c6931cf74"}
Oct 30 01:00:16.435249 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:16.435165    6099 scope.go:122] "RemoveContainer" containerID="21b2b68e62de806c328d1946643254519cab3e76f988257b0e72908f9ab25477"
Oct 30 01:00:16.436512 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:16.436486    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4"
Oct 30 01:00:16.436634 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:16.436524    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4" event={"ID":"0e190a98-17fe-46f6-adcf-34223eaf820e","Type":"ContainerDied","Data":"ac59ecf0ce90b52bd1f24fe98fb3c11961c07b66e0d7d9de08145af46595bdaf"}
Oct 30 01:00:16.436818 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:16.436799    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:16.438423 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:16.438113    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:00:16.477392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:16.477370    6099 scope.go:122] "RemoveContainer" containerID="28a44dc7433d5dd5158e619d7485433c6e041166705909aea99f6cb0bec6ea48"
Oct 30 01:00:16.540362 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:16.540299    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4"]
Oct 30 01:00:16.544526 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:16.544501    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a32412-6b6cc7db6c-xfbc4"]
Oct 30 01:00:16.572655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:16.572628    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m"]
Oct 30 01:00:16.576221 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:16.576200    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a32412-ccf56585d-ns54m"]
Oct 30 01:00:16.700022 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:16.699993    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:00:17.254542 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:17.254505    6099 scope.go:122] "RemoveContainer" containerID="f20075a6f5d98bfae69954722ec529e46eb1a8d32b4b163ce4780e63dc036d85"
Oct 30 01:00:17.254741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:17.254690    6099 scope.go:122] "RemoveContainer" containerID="2deb607e40577b597147bf69eb9084f857e641f88b8b499bb8b6e6be47e8b8db"
Oct 30 01:00:17.450066 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:17.449992    6099 generic.go:356] "Generic (PLEG): container finished" podID="0e2294af-beb7-4d49-8cd8-4abd0c04a40a" containerID="e7dd16bcc4154912f278fba96e65c7ce68d8998a9c0e4e9b8594d32c78f5cced" exitCode=60
Oct 30 01:00:17.450520 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:17.450074    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl" event={"ID":"0e2294af-beb7-4d49-8cd8-4abd0c04a40a","Type":"ContainerDied","Data":"e7dd16bcc4154912f278fba96e65c7ce68d8998a9c0e4e9b8594d32c78f5cced"}
Oct 30 01:00:17.450520 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:17.450143    6099 scope.go:122] "RemoveContainer" containerID="fdfe9116ab85773c9d0fdf56b577a257ad05d0c5df76babeac5f96f334495b9d"
Oct 30 01:00:17.450772 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:17.450754    6099 scope.go:122] "RemoveContainer" containerID="e7dd16bcc4154912f278fba96e65c7ce68d8998a9c0e4e9b8594d32c78f5cced"
Oct 30 01:00:17.451178 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:17.451146    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888700-9p4dl_odh-ai-gateway-infra(0e2294af-beb7-4d49-8cd8-4abd0c04a40a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl" podUID="0e2294af-beb7-4d49-8cd8-4abd0c04a40a"
Oct 30 01:00:17.452766 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:17.452744    6099 generic.go:356] "Generic (PLEG): container finished" podID="3365bf0b-a96c-429e-8565-faa44fef8ee0" containerID="a2dffcf9e780f7f25f2a88df53d4a7f24345f719a9936412b915df508d4ef5ef" exitCode=60
Oct 30 01:00:17.452882 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:17.452818    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl" event={"ID":"3365bf0b-a96c-429e-8565-faa44fef8ee0","Type":"ContainerDied","Data":"a2dffcf9e780f7f25f2a88df53d4a7f24345f719a9936412b915df508d4ef5ef"}
Oct 30 01:00:17.453217 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:17.453202    6099 scope.go:122] "RemoveContainer" containerID="a2dffcf9e780f7f25f2a88df53d4a7f24345f719a9936412b915df508d4ef5ef"
Oct 30 01:00:17.453580 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:17.453561    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl_odh-ai-gateway-infra(3365bf0b-a96c-429e-8565-faa44fef8ee0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl" podUID="3365bf0b-a96c-429e-8565-faa44fef8ee0"
Oct 30 01:00:17.457493 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:17.457469    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k" event={"ID":"e23eea04-ac89-4fa4-bb29-f38d655df0d7","Type":"ContainerStarted","Data":"354ae02f181ea8c23ae98fc0f66e9e27cc718e729da1fa8eb145aaf32cbee8b3"}
Oct 30 01:00:17.466951 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:17.466933    6099 scope.go:122] "RemoveContainer" containerID="2f86196dc2acb9a1ca5a79d3cd75b1b22ee6257de4798d53c28e593e6855b96a"
Oct 30 01:00:18.014003 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.013976    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 01:00:18.040311 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.040288    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 01:00:18.105175 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.105141    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-envoy\") pod \"19e9d22c-b2b4-4795-be11-ff2d98383a92\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") "
Oct 30 01:00:18.105348 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.105189    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-workload-certs\") pod \"6474c172-c1ba-43d1-a965-242e6b93a3a2\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") "
Oct 30 01:00:18.105348 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.105305    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6474c172-c1ba-43d1-a965-242e6b93a3a2-kube-api-access-l8lmq\" (UniqueName: \"kubernetes.io/projected/6474c172-c1ba-43d1-a965-242e6b93a3a2-kube-api-access-l8lmq\") pod \"6474c172-c1ba-43d1-a965-242e6b93a3a2\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") "
Oct 30 01:00:18.105348 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.105346    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-podinfo\") pod \"19e9d22c-b2b4-4795-be11-ff2d98383a92\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") "
Oct 30 01:00:18.105507 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.105376    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6474c172-c1ba-43d1-a965-242e6b93a3a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6474c172-c1ba-43d1-a965-242e6b93a3a2-istiod-ca-cert\") pod \"6474c172-c1ba-43d1-a965-242e6b93a3a2\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") "
Oct 30 01:00:18.105891 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.105867    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-envoy\") pod \"6474c172-c1ba-43d1-a965-242e6b93a3a2\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") "
Oct 30 01:00:18.106011 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.105541    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-workload-certs" pod "6474c172-c1ba-43d1-a965-242e6b93a3a2" (UID: "6474c172-c1ba-43d1-a965-242e6b93a3a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:18.106011 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.105803    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6474c172-c1ba-43d1-a965-242e6b93a3a2-istiod-ca-cert" pod "6474c172-c1ba-43d1-a965-242e6b93a3a2" (UID: "6474c172-c1ba-43d1-a965-242e6b93a3a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:18.106011 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.106006    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-credential-socket\") pod \"19e9d22c-b2b4-4795-be11-ff2d98383a92\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") "
Oct 30 01:00:18.106820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.106039    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-data\") pod \"19e9d22c-b2b4-4795-be11-ff2d98383a92\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") "
Oct 30 01:00:18.106820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.106075    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-workload-socket\") pod \"6474c172-c1ba-43d1-a965-242e6b93a3a2\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") "
Oct 30 01:00:18.106820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.106186    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-token\" (UniqueName: \"kubernetes.io/projected/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-token\") pod \"19e9d22c-b2b4-4795-be11-ff2d98383a92\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") "
Oct 30 01:00:18.106820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.106228    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-workload-socket\") pod \"19e9d22c-b2b4-4795-be11-ff2d98383a92\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") "
Oct 30 01:00:18.106820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.106355    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-credential-socket" pod "19e9d22c-b2b4-4795-be11-ff2d98383a92" (UID: "19e9d22c-b2b4-4795-be11-ff2d98383a92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:18.106820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.106387    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-data\") pod \"6474c172-c1ba-43d1-a965-242e6b93a3a2\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") "
Oct 30 01:00:18.106820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.106416    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-workload-socket" pod "6474c172-c1ba-43d1-a965-242e6b93a3a2" (UID: "6474c172-c1ba-43d1-a965-242e6b93a3a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:18.106820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.106452    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e9d22c-b2b4-4795-be11-ff2d98383a92-kube-api-access-vfsm4\" (UniqueName: \"kubernetes.io/projected/19e9d22c-b2b4-4795-be11-ff2d98383a92-kube-api-access-vfsm4\") pod \"19e9d22c-b2b4-4795-be11-ff2d98383a92\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") "
Oct 30 01:00:18.106820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.106519    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-token\" (UniqueName: \"kubernetes.io/projected/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-token\") pod \"6474c172-c1ba-43d1-a965-242e6b93a3a2\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") "
Oct 30 01:00:18.106820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.106557    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19e9d22c-b2b4-4795-be11-ff2d98383a92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e9d22c-b2b4-4795-be11-ff2d98383a92-istiod-ca-cert\") pod \"19e9d22c-b2b4-4795-be11-ff2d98383a92\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") "
Oct 30 01:00:18.106820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.106601    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-podinfo\") pod \"6474c172-c1ba-43d1-a965-242e6b93a3a2\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") "
Oct 30 01:00:18.106820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.106616    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-workload-socket" pod "19e9d22c-b2b4-4795-be11-ff2d98383a92" (UID: "19e9d22c-b2b4-4795-be11-ff2d98383a92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:18.106820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.106639    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-credential-socket\") pod \"6474c172-c1ba-43d1-a965-242e6b93a3a2\" (UID: \"6474c172-c1ba-43d1-a965-242e6b93a3a2\") "
Oct 30 01:00:18.106820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.106673    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-workload-certs\") pod \"19e9d22c-b2b4-4795-be11-ff2d98383a92\" (UID: \"19e9d22c-b2b4-4795-be11-ff2d98383a92\") "
Oct 30 01:00:18.107610 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.107109    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-data" pod "19e9d22c-b2b4-4795-be11-ff2d98383a92" (UID: "19e9d22c-b2b4-4795-be11-ff2d98383a92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:18.107610 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.107315    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-workload-certs" pod "19e9d22c-b2b4-4795-be11-ff2d98383a92" (UID: "19e9d22c-b2b4-4795-be11-ff2d98383a92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:18.107610 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.107572    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.107610 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.107594    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6474c172-c1ba-43d1-a965-242e6b93a3a2-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.108352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.107636    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.108352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.107648    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e9d22c-b2b4-4795-be11-ff2d98383a92-istiod-ca-cert" pod "19e9d22c-b2b4-4795-be11-ff2d98383a92" (UID: "19e9d22c-b2b4-4795-be11-ff2d98383a92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:18.108352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.107653    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.108352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.107690    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.108352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.107722    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.108352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.107738    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.108352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.107996    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-data" pod "6474c172-c1ba-43d1-a965-242e6b93a3a2" (UID: "6474c172-c1ba-43d1-a965-242e6b93a3a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:18.108352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.108013    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-credential-socket" pod "6474c172-c1ba-43d1-a965-242e6b93a3a2" (UID: "6474c172-c1ba-43d1-a965-242e6b93a3a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:18.108924 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.108876    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-envoy" pod "19e9d22c-b2b4-4795-be11-ff2d98383a92" (UID: "19e9d22c-b2b4-4795-be11-ff2d98383a92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:18.110224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.110145    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-token" pod "19e9d22c-b2b4-4795-be11-ff2d98383a92" (UID: "19e9d22c-b2b4-4795-be11-ff2d98383a92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:18.110327 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.110237    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-podinfo" pod "19e9d22c-b2b4-4795-be11-ff2d98383a92" (UID: "19e9d22c-b2b4-4795-be11-ff2d98383a92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:00:18.110327 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.110248    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6474c172-c1ba-43d1-a965-242e6b93a3a2-kube-api-access-l8lmq" pod "6474c172-c1ba-43d1-a965-242e6b93a3a2" (UID: "6474c172-c1ba-43d1-a965-242e6b93a3a2"). InnerVolumeSpecName "kube-api-access-l8lmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:18.110533 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.110505    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-podinfo" pod "6474c172-c1ba-43d1-a965-242e6b93a3a2" (UID: "6474c172-c1ba-43d1-a965-242e6b93a3a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:00:18.110865 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.110838    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-token" pod "6474c172-c1ba-43d1-a965-242e6b93a3a2" (UID: "6474c172-c1ba-43d1-a965-242e6b93a3a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:18.111431 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.111368    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e9d22c-b2b4-4795-be11-ff2d98383a92-kube-api-access-vfsm4" pod "19e9d22c-b2b4-4795-be11-ff2d98383a92" (UID: "19e9d22c-b2b4-4795-be11-ff2d98383a92"). InnerVolumeSpecName "kube-api-access-vfsm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:18.112134 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.112107    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-envoy" pod "6474c172-c1ba-43d1-a965-242e6b93a3a2" (UID: "6474c172-c1ba-43d1-a965-242e6b93a3a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:18.209079 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.209043    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.209079 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.209078    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.209079 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.209087    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfsm4\" (UniqueName: \"kubernetes.io/projected/19e9d22c-b2b4-4795-be11-ff2d98383a92-kube-api-access-vfsm4\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.209312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.209097    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.209312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.209105    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e9d22c-b2b4-4795-be11-ff2d98383a92-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.209312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.209113    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.209312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.209120    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.209312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.209129    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.209312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.209139    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8lmq\" (UniqueName: \"kubernetes.io/projected/6474c172-c1ba-43d1-a965-242e6b93a3a2-kube-api-access-l8lmq\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.209312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.209150    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e9d22c-b2b4-4795-be11-ff2d98383a92-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.209312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.209158    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6474c172-c1ba-43d1-a965-242e6b93a3a2-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.264170 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.264093    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e190a98-17fe-46f6-adcf-34223eaf820e" path="/var/lib/kubelet/pods/0e190a98-17fe-46f6-adcf-34223eaf820e/volumes"
Oct 30 01:00:18.264741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.264721    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20857824-d001-4494-97f3-ff58b868c3c2" path="/var/lib/kubelet/pods/20857824-d001-4494-97f3-ff58b868c3c2/volumes"
Oct 30 01:00:18.267465 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.266514    6099 scope.go:122] "RemoveContainer" containerID="97bdba38dd333e0e077dc6bc4ddbad1d9eaae9e898fc56e46b79b4ecafa9c486"
Oct 30 01:00:18.474745 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.474694    6099 generic.go:356] "Generic (PLEG): container finished" podID="fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9" containerID="e579730b16a3ed3e339bfc46538fd93f4ea94bdd195693819753922f59395fe6" exitCode=60
Oct 30 01:00:18.475174 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.474765    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2" event={"ID":"fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9","Type":"ContainerDied","Data":"e579730b16a3ed3e339bfc46538fd93f4ea94bdd195693819753922f59395fe6"}
Oct 30 01:00:18.475174 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.474839    6099 scope.go:122] "RemoveContainer" containerID="f20075a6f5d98bfae69954722ec529e46eb1a8d32b4b163ce4780e63dc036d85"
Oct 30 01:00:18.475332 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.475310    6099 scope.go:122] "RemoveContainer" containerID="e579730b16a3ed3e339bfc46538fd93f4ea94bdd195693819753922f59395fe6"
Oct 30 01:00:18.475785 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:18.475760    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2_odh-ai-gateway-infra(fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2" podUID="fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9"
Oct 30 01:00:18.477600 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.477576    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw" event={"ID":"9c86d85f-10ed-4300-98d7-e6934451d12d","Type":"ContainerStarted","Data":"e36a0668d7673488b166dd7dbb2bd4459d3d38c4d7ebf9ec5fed238bad91c211"}
Oct 30 01:00:18.478027 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.478009    6099 scope.go:122] "RemoveContainer" containerID="e36a0668d7673488b166dd7dbb2bd4459d3d38c4d7ebf9ec5fed238bad91c211"
Oct 30 01:00:18.478419 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:18.478395    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw_odh-ai-gateway-infra(9c86d85f-10ed-4300-98d7-e6934451d12d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw" podUID="9c86d85f-10ed-4300-98d7-e6934451d12d"
Oct 30 01:00:18.479626 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.479607    6099 generic.go:356] "Generic (PLEG): container finished" podID="e23eea04-ac89-4fa4-bb29-f38d655df0d7" containerID="354ae02f181ea8c23ae98fc0f66e9e27cc718e729da1fa8eb145aaf32cbee8b3" exitCode=60
Oct 30 01:00:18.479752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.479730    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k" event={"ID":"e23eea04-ac89-4fa4-bb29-f38d655df0d7","Type":"ContainerDied","Data":"354ae02f181ea8c23ae98fc0f66e9e27cc718e729da1fa8eb145aaf32cbee8b3"}
Oct 30 01:00:18.480159 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.480140    6099 scope.go:122] "RemoveContainer" containerID="354ae02f181ea8c23ae98fc0f66e9e27cc718e729da1fa8eb145aaf32cbee8b3"
Oct 30 01:00:18.480528 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:18.480505    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k_odh-ai-gateway-infra(e23eea04-ac89-4fa4-bb29-f38d655df0d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k" podUID="e23eea04-ac89-4fa4-bb29-f38d655df0d7"
Oct 30 01:00:18.481691 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.481671    6099 generic.go:356] "Generic (PLEG): container finished" podID="19e9d22c-b2b4-4795-be11-ff2d98383a92" containerID="8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853" exitCode=0
Oct 30 01:00:18.481791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.481741    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"
Oct 30 01:00:18.481791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.481742    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw" event={"ID":"19e9d22c-b2b4-4795-be11-ff2d98383a92","Type":"ContainerDied","Data":"8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853"}
Oct 30 01:00:18.481917 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.481785    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw" event={"ID":"19e9d22c-b2b4-4795-be11-ff2d98383a92","Type":"ContainerDied","Data":"c3e728f58edd178b4e3c9c0d40c226b9db772799c90b0b478664f719802d0798"}
Oct 30 01:00:18.485215 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.485196    6099 generic.go:356] "Generic (PLEG): container finished" podID="6474c172-c1ba-43d1-a965-242e6b93a3a2" containerID="59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764" exitCode=0
Oct 30 01:00:18.485297 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.485242    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"
Oct 30 01:00:18.485297 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.485280    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj" event={"ID":"6474c172-c1ba-43d1-a965-242e6b93a3a2","Type":"ContainerDied","Data":"59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764"}
Oct 30 01:00:18.485402 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.485327    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj" event={"ID":"6474c172-c1ba-43d1-a965-242e6b93a3a2","Type":"ContainerDied","Data":"e25508c55536b130cb5c39586f2260eb079617e7dc7bcd3233bd6068ef0d75b6"}
Oct 30 01:00:18.510833 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.510807    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl"]
Oct 30 01:00:18.531476 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.531452    6099 scope.go:122] "RemoveContainer" containerID="d67ec0d59179b350dccaa52c3d47aa626b48b5c1b1d4af70bbcbd1fa8e9fd87d"
Oct 30 01:00:18.552532 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.552498    6099 scope.go:122] "RemoveContainer" containerID="8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853"
Oct 30 01:00:18.557244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.557216    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl"]
Oct 30 01:00:18.570797 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.570773    6099 scope.go:122] "RemoveContainer" containerID="485ef597e39d43f5e309aec055ac6004c7ce23ed9b99c38bbb4b207671d7e6dc"
Oct 30 01:00:18.590302 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.590272    6099 scope.go:122] "RemoveContainer" containerID="8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853"
Oct 30 01:00:18.590666 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:18.590639    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853\": container with ID starting with 8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853 not found: ID does not exist" containerID="8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853"
Oct 30 01:00:18.590823 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.590674    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853"} err="failed to get container status \"8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853\": rpc error: code = NotFound desc = could not find container \"8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853\": container with ID starting with 8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853 not found: ID does not exist"
Oct 30 01:00:18.590823 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.590700    6099 scope.go:122] "RemoveContainer" containerID="485ef597e39d43f5e309aec055ac6004c7ce23ed9b99c38bbb4b207671d7e6dc"
Oct 30 01:00:18.591079 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:18.591006    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"485ef597e39d43f5e309aec055ac6004c7ce23ed9b99c38bbb4b207671d7e6dc\": container with ID starting with 485ef597e39d43f5e309aec055ac6004c7ce23ed9b99c38bbb4b207671d7e6dc not found: ID does not exist" containerID="485ef597e39d43f5e309aec055ac6004c7ce23ed9b99c38bbb4b207671d7e6dc"
Oct 30 01:00:18.591079 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.591040    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"485ef597e39d43f5e309aec055ac6004c7ce23ed9b99c38bbb4b207671d7e6dc"} err="failed to get container status \"485ef597e39d43f5e309aec055ac6004c7ce23ed9b99c38bbb4b207671d7e6dc\": rpc error: code = NotFound desc = could not find container \"485ef597e39d43f5e309aec055ac6004c7ce23ed9b99c38bbb4b207671d7e6dc\": container with ID starting with 485ef597e39d43f5e309aec055ac6004c7ce23ed9b99c38bbb4b207671d7e6dc not found: ID does not exist"
Oct 30 01:00:18.591079 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.591060    6099 scope.go:122] "RemoveContainer" containerID="59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764"
Oct 30 01:00:18.595926 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.595900    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k"]
Oct 30 01:00:18.636804 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.636749    6099 scope.go:122] "RemoveContainer" containerID="59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764"
Oct 30 01:00:18.638185 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:18.638123    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764\": container with ID starting with 59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764 not found: ID does not exist" containerID="59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764"
Oct 30 01:00:18.638185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.638162    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764"} err="failed to get container status \"59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764\": rpc error: code = NotFound desc = could not find container \"59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764\": container with ID starting with 59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764 not found: ID does not exist"
Oct 30 01:00:18.734244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.733291    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"]
Oct 30 01:00:18.739219 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.738921    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-dj2mj"]
Oct 30 01:00:18.744637 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.744614    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl"
Oct 30 01:00:18.772683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.772656    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"]
Oct 30 01:00:18.778621 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.778596    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-73b32e-openshift-default-5d4d7cb979-xz9qw"]
Oct 30 01:00:18.782430 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.782383    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl"
Oct 30 01:00:18.820693 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.820656    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3365bf0b-a96c-429e-8565-faa44fef8ee0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3365bf0b-a96c-429e-8565-faa44fef8ee0-maas-api-service-ca\") pod \"3365bf0b-a96c-429e-8565-faa44fef8ee0\" (UID: \"3365bf0b-a96c-429e-8565-faa44fef8ee0\") "
Oct 30 01:00:18.820871 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.820817    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e2294af-beb7-4d49-8cd8-4abd0c04a40a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e2294af-beb7-4d49-8cd8-4abd0c04a40a-maas-api-service-ca\") pod \"0e2294af-beb7-4d49-8cd8-4abd0c04a40a\" (UID: \"0e2294af-beb7-4d49-8cd8-4abd0c04a40a\") "
Oct 30 01:00:18.820871 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.820841    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e2294af-beb7-4d49-8cd8-4abd0c04a40a-kube-api-access-87z9f\" (UniqueName: \"kubernetes.io/projected/0e2294af-beb7-4d49-8cd8-4abd0c04a40a-kube-api-access-87z9f\") pod \"0e2294af-beb7-4d49-8cd8-4abd0c04a40a\" (UID: \"0e2294af-beb7-4d49-8cd8-4abd0c04a40a\") "
Oct 30 01:00:18.820995 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.820960    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3365bf0b-a96c-429e-8565-faa44fef8ee0-kube-api-access-6bf6w\" (UniqueName: \"kubernetes.io/projected/3365bf0b-a96c-429e-8565-faa44fef8ee0-kube-api-access-6bf6w\") pod \"3365bf0b-a96c-429e-8565-faa44fef8ee0\" (UID: \"3365bf0b-a96c-429e-8565-faa44fef8ee0\") "
Oct 30 01:00:18.821058 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.820990    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3365bf0b-a96c-429e-8565-faa44fef8ee0-maas-api-service-ca" pod "3365bf0b-a96c-429e-8565-faa44fef8ee0" (UID: "3365bf0b-a96c-429e-8565-faa44fef8ee0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:18.821227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.821207    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e2294af-beb7-4d49-8cd8-4abd0c04a40a-maas-api-service-ca" pod "0e2294af-beb7-4d49-8cd8-4abd0c04a40a" (UID: "0e2294af-beb7-4d49-8cd8-4abd0c04a40a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:18.821689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.821671    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3365bf0b-a96c-429e-8565-faa44fef8ee0-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.821689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.821690    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e2294af-beb7-4d49-8cd8-4abd0c04a40a-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.823256 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.823222    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3365bf0b-a96c-429e-8565-faa44fef8ee0-kube-api-access-6bf6w" pod "3365bf0b-a96c-429e-8565-faa44fef8ee0" (UID: "3365bf0b-a96c-429e-8565-faa44fef8ee0"). InnerVolumeSpecName "kube-api-access-6bf6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:18.823328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.823314    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2294af-beb7-4d49-8cd8-4abd0c04a40a-kube-api-access-87z9f" pod "0e2294af-beb7-4d49-8cd8-4abd0c04a40a" (UID: "0e2294af-beb7-4d49-8cd8-4abd0c04a40a"). InnerVolumeSpecName "kube-api-access-87z9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:18.923440 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.923397    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87z9f\" (UniqueName: \"kubernetes.io/projected/0e2294af-beb7-4d49-8cd8-4abd0c04a40a-kube-api-access-87z9f\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.923440 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:18.923429    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bf6w\" (UniqueName: \"kubernetes.io/projected/3365bf0b-a96c-429e-8565-faa44fef8ee0-kube-api-access-6bf6w\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:19.255107 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.255071    6099 scope.go:122] "RemoveContainer" containerID="0769681754cb16ec2e57cdc67971934829ee36edf542e457e882443a07a98b94"
Oct 30 01:00:19.255293 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.255207    6099 scope.go:122] "RemoveContainer" containerID="4a1dd6cb9cf50a6d1c5442604282d5504e508652d61ad1f2a648c2ed93add066"
Oct 30 01:00:19.499440 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.499404    6099 generic.go:356] "Generic (PLEG): container finished" podID="f378df0c-bde8-4326-ade3-7c4ecd956cb3" containerID="b664363ee8b35afbadfed7c112f3aae8dccfeb532af138ec75bbd148a9d790f1" exitCode=60
Oct 30 01:00:19.499867 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.499480    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz" event={"ID":"f378df0c-bde8-4326-ade3-7c4ecd956cb3","Type":"ContainerDied","Data":"b664363ee8b35afbadfed7c112f3aae8dccfeb532af138ec75bbd148a9d790f1"}
Oct 30 01:00:19.499867 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.499564    6099 scope.go:122] "RemoveContainer" containerID="97bdba38dd333e0e077dc6bc4ddbad1d9eaae9e898fc56e46b79b4ecafa9c486"
Oct 30 01:00:19.500139 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.500117    6099 scope.go:122] "RemoveContainer" containerID="b664363ee8b35afbadfed7c112f3aae8dccfeb532af138ec75bbd148a9d790f1"
Oct 30 01:00:19.500593 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:19.500564    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz_odh-ai-gateway-infra(f378df0c-bde8-4326-ade3-7c4ecd956cb3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz" podUID="f378df0c-bde8-4326-ade3-7c4ecd956cb3"
Oct 30 01:00:19.504991 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.504962    6099 generic.go:356] "Generic (PLEG): container finished" podID="9c86d85f-10ed-4300-98d7-e6934451d12d" containerID="e36a0668d7673488b166dd7dbb2bd4459d3d38c4d7ebf9ec5fed238bad91c211" exitCode=60
Oct 30 01:00:19.505121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.505005    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw" event={"ID":"9c86d85f-10ed-4300-98d7-e6934451d12d","Type":"ContainerDied","Data":"e36a0668d7673488b166dd7dbb2bd4459d3d38c4d7ebf9ec5fed238bad91c211"}
Oct 30 01:00:19.505563 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.505542    6099 scope.go:122] "RemoveContainer" containerID="e36a0668d7673488b166dd7dbb2bd4459d3d38c4d7ebf9ec5fed238bad91c211"
Oct 30 01:00:19.506164 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:19.506139    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw_odh-ai-gateway-infra(9c86d85f-10ed-4300-98d7-e6934451d12d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw" podUID="9c86d85f-10ed-4300-98d7-e6934451d12d"
Oct 30 01:00:19.514638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.514475    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl" event={"ID":"0e2294af-beb7-4d49-8cd8-4abd0c04a40a","Type":"ContainerDied","Data":"2a3c7b7be0184413ab4e2b3e2dfbe9c2b8b31adcd1c8d4b03fa98a9025caf44b"}
Oct 30 01:00:19.514638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.514575    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl"
Oct 30 01:00:19.527668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.527642    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl" event={"ID":"3365bf0b-a96c-429e-8565-faa44fef8ee0","Type":"ContainerDied","Data":"2cf80dc01dcee76806582442e72d51c4a102ec44f94a03eb64e9df174f648544"}
Oct 30 01:00:19.527817 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.527784    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl"
Oct 30 01:00:19.535689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.535212    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2"]
Oct 30 01:00:19.537942 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.537924    6099 scope.go:122] "RemoveContainer" containerID="2deb607e40577b597147bf69eb9084f857e641f88b8b499bb8b6e6be47e8b8db"
Oct 30 01:00:19.571468 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.571395    6099 scope.go:122] "RemoveContainer" containerID="e7dd16bcc4154912f278fba96e65c7ce68d8998a9c0e4e9b8594d32c78f5cced"
Oct 30 01:00:19.589036 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.588995    6099 scope.go:122] "RemoveContainer" containerID="a2dffcf9e780f7f25f2a88df53d4a7f24345f719a9936412b915df508d4ef5ef"
Oct 30 01:00:19.651751 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.651718    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl"]
Oct 30 01:00:19.661339 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.661313    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2b4105-29888700-79bgl"]
Oct 30 01:00:19.676185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.676154    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw"]
Oct 30 01:00:19.703666 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.703645    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl"]
Oct 30 01:00:19.718481 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.718452    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888700-9p4dl"]
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.765830    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl"]
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778010    6099 cpu_manager.go:414] "containerMap: removing container" podUID="19e9d22c-b2b4-4795-be11-ff2d98383a92" containerName="istio-proxy"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778033    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e9d22c-b2b4-4795-be11-ff2d98383a92" containerName="istio-proxy"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778060    6099 cpu_manager.go:414] "containerMap: removing container" podUID="0e2294af-beb7-4d49-8cd8-4abd0c04a40a" containerName="cleanup"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778067    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e2294af-beb7-4d49-8cd8-4abd0c04a40a" containerName="cleanup"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778080    6099 cpu_manager.go:414] "containerMap: removing container" podUID="0e2294af-beb7-4d49-8cd8-4abd0c04a40a" containerName="cleanup"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778087    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e2294af-beb7-4d49-8cd8-4abd0c04a40a" containerName="cleanup"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778106    6099 cpu_manager.go:414] "containerMap: removing container" podUID="3365bf0b-a96c-429e-8565-faa44fef8ee0" containerName="cleanup"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778113    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3365bf0b-a96c-429e-8565-faa44fef8ee0" containerName="cleanup"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778137    6099 cpu_manager.go:414] "containerMap: removing container" podUID="20857824-d001-4494-97f3-ff58b868c3c2" containerName="payload-processing"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778144    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20857824-d001-4494-97f3-ff58b868c3c2" containerName="payload-processing"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778161    6099 cpu_manager.go:414] "containerMap: removing container" podUID="0e2294af-beb7-4d49-8cd8-4abd0c04a40a" containerName="cleanup"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778169    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e2294af-beb7-4d49-8cd8-4abd0c04a40a" containerName="cleanup"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778185    6099 cpu_manager.go:414] "containerMap: removing container" podUID="3365bf0b-a96c-429e-8565-faa44fef8ee0" containerName="cleanup"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778192    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3365bf0b-a96c-429e-8565-faa44fef8ee0" containerName="cleanup"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778228    6099 cpu_manager.go:414] "containerMap: removing container" podUID="0e190a98-17fe-46f6-adcf-34223eaf820e" containerName="payload-pre-processing"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778235    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e190a98-17fe-46f6-adcf-34223eaf820e" containerName="payload-pre-processing"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778248    6099 cpu_manager.go:414] "containerMap: removing container" podUID="6474c172-c1ba-43d1-a965-242e6b93a3a2" containerName="istio-proxy"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778254    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6474c172-c1ba-43d1-a965-242e6b93a3a2" containerName="istio-proxy"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778273    6099 cpu_manager.go:414] "containerMap: removing container" podUID="3365bf0b-a96c-429e-8565-faa44fef8ee0" containerName="cleanup"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778279    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3365bf0b-a96c-429e-8565-faa44fef8ee0" containerName="cleanup"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778431    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="20857824-d001-4494-97f3-ff58b868c3c2" containerName="payload-processing"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778443    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e2294af-beb7-4d49-8cd8-4abd0c04a40a" containerName="cleanup"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778455    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e9d22c-b2b4-4795-be11-ff2d98383a92" containerName="istio-proxy"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778465    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3365bf0b-a96c-429e-8565-faa44fef8ee0" containerName="cleanup"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778477    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e190a98-17fe-46f6-adcf-34223eaf820e" containerName="payload-pre-processing"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778488    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6474c172-c1ba-43d1-a965-242e6b93a3a2" containerName="istio-proxy"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778498    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3365bf0b-a96c-429e-8565-faa44fef8ee0" containerName="cleanup"
Oct 30 01:00:19.781762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.778512    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e2294af-beb7-4d49-8cd8-4abd0c04a40a" containerName="cleanup"
Oct 30 01:00:19.787345 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.786587    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl"
Oct 30 01:00:19.792231 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.792204    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl"]
Oct 30 01:00:19.804667 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.804644    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-780077-dockercfg-2mzcr\""
Oct 30 01:00:19.804896 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.804881    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-780077\""
Oct 30 01:00:19.832329 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.829061    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh"]
Oct 30 01:00:19.832329 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.830731    6099 cpu_manager.go:414] "containerMap: removing container" podUID="19e9d22c-b2b4-4795-be11-ff2d98383a92" containerName="istio-proxy"
Oct 30 01:00:19.832329 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.830756    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e9d22c-b2b4-4795-be11-ff2d98383a92" containerName="istio-proxy"
Oct 30 01:00:19.832329 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.831023    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e2294af-beb7-4d49-8cd8-4abd0c04a40a" containerName="cleanup"
Oct 30 01:00:19.832329 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.831048    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e9d22c-b2b4-4795-be11-ff2d98383a92" containerName="istio-proxy"
Oct 30 01:00:19.832329 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.831061    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3365bf0b-a96c-429e-8565-faa44fef8ee0" containerName="cleanup"
Oct 30 01:00:19.838597 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.838427    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnb7k\" (UniqueName: \"kubernetes.io/projected/2e720881-a65b-4e3b-95c1-e88b04a6c88a-kube-api-access-lnb7k\") pod \"payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl\" (UID: \"2e720881-a65b-4e3b-95c1-e88b04a6c88a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl"
Oct 30 01:00:19.841296 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.841232    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e720881-a65b-4e3b-95c1-e88b04a6c88a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl\" (UID: \"2e720881-a65b-4e3b-95c1-e88b04a6c88a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl"
Oct 30 01:00:19.979607 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.948361    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e720881-a65b-4e3b-95c1-e88b04a6c88a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl\" (UID: \"2e720881-a65b-4e3b-95c1-e88b04a6c88a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl"
Oct 30 01:00:19.981378 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.981336    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnb7k\" (UniqueName: \"kubernetes.io/projected/2e720881-a65b-4e3b-95c1-e88b04a6c88a-kube-api-access-lnb7k\") pod \"payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl\" (UID: \"2e720881-a65b-4e3b-95c1-e88b04a6c88a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl"
Oct 30 01:00:19.988680 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:19.988604    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e720881-a65b-4e3b-95c1-e88b04a6c88a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl\" (UID: \"2e720881-a65b-4e3b-95c1-e88b04a6c88a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl"
Oct 30 01:00:20.013890 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.013859    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnb7k\" (UniqueName: \"kubernetes.io/projected/2e720881-a65b-4e3b-95c1-e88b04a6c88a-kube-api-access-lnb7k\") pod \"payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl\" (UID: \"2e720881-a65b-4e3b-95c1-e88b04a6c88a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl"
Oct 30 01:00:20.019902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.018856    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k"
Oct 30 01:00:20.052936 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.044688    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh"]
Oct 30 01:00:20.053216 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.047836    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh"
Oct 30 01:00:20.062409 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.062383    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2"
Oct 30 01:00:20.113496 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.113466    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl"
Oct 30 01:00:20.185474 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.185431    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9-maas-api-service-ca\") pod \"fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9\" (UID: \"fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9\") "
Oct 30 01:00:20.185643 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.185572    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23eea04-ac89-4fa4-bb29-f38d655df0d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23eea04-ac89-4fa4-bb29-f38d655df0d7-maas-api-service-ca\") pod \"e23eea04-ac89-4fa4-bb29-f38d655df0d7\" (UID: \"e23eea04-ac89-4fa4-bb29-f38d655df0d7\") "
Oct 30 01:00:20.185739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.185678    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23eea04-ac89-4fa4-bb29-f38d655df0d7-kube-api-access-mzhk4\" (UniqueName: \"kubernetes.io/projected/e23eea04-ac89-4fa4-bb29-f38d655df0d7-kube-api-access-mzhk4\") pod \"e23eea04-ac89-4fa4-bb29-f38d655df0d7\" (UID: \"e23eea04-ac89-4fa4-bb29-f38d655df0d7\") "
Oct 30 01:00:20.185915 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.185898    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9-kube-api-access-tw2pm\" (UniqueName: \"kubernetes.io/projected/fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9-kube-api-access-tw2pm\") pod \"fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9\" (UID: \"fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9\") "
Oct 30 01:00:20.186182 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.186160    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk5zj\" (UniqueName: \"kubernetes.io/projected/65fe9203-ea37-4a79-9990-5ca46ceaccd0-kube-api-access-fk5zj\") pod \"payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh\" (UID: \"65fe9203-ea37-4a79-9990-5ca46ceaccd0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh"
Oct 30 01:00:20.186399 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.186383    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65fe9203-ea37-4a79-9990-5ca46ceaccd0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh\" (UID: \"65fe9203-ea37-4a79-9990-5ca46ceaccd0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh"
Oct 30 01:00:20.188169 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.188093    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23eea04-ac89-4fa4-bb29-f38d655df0d7-maas-api-service-ca" pod "e23eea04-ac89-4fa4-bb29-f38d655df0d7" (UID: "e23eea04-ac89-4fa4-bb29-f38d655df0d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:20.188341 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.188233    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9-maas-api-service-ca" pod "fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9" (UID: "fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:20.189565 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.189531    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9-kube-api-access-tw2pm" pod "fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9" (UID: "fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9"). InnerVolumeSpecName "kube-api-access-tw2pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:20.190060 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.190038    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23eea04-ac89-4fa4-bb29-f38d655df0d7-kube-api-access-mzhk4" pod "e23eea04-ac89-4fa4-bb29-f38d655df0d7" (UID: "e23eea04-ac89-4fa4-bb29-f38d655df0d7"). InnerVolumeSpecName "kube-api-access-mzhk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:20.264770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.264679    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e2294af-beb7-4d49-8cd8-4abd0c04a40a" path="/var/lib/kubelet/pods/0e2294af-beb7-4d49-8cd8-4abd0c04a40a/volumes"
Oct 30 01:00:20.265308 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.265284    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19e9d22c-b2b4-4795-be11-ff2d98383a92" path="/var/lib/kubelet/pods/19e9d22c-b2b4-4795-be11-ff2d98383a92/volumes"
Oct 30 01:00:20.265844 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.265824    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3365bf0b-a96c-429e-8565-faa44fef8ee0" path="/var/lib/kubelet/pods/3365bf0b-a96c-429e-8565-faa44fef8ee0/volumes"
Oct 30 01:00:20.266815 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.266796    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6474c172-c1ba-43d1-a965-242e6b93a3a2" path="/var/lib/kubelet/pods/6474c172-c1ba-43d1-a965-242e6b93a3a2/volumes"
Oct 30 01:00:20.288310 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.288262    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk5zj\" (UniqueName: \"kubernetes.io/projected/65fe9203-ea37-4a79-9990-5ca46ceaccd0-kube-api-access-fk5zj\") pod \"payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh\" (UID: \"65fe9203-ea37-4a79-9990-5ca46ceaccd0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh"
Oct 30 01:00:20.289091 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.288690    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65fe9203-ea37-4a79-9990-5ca46ceaccd0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh\" (UID: \"65fe9203-ea37-4a79-9990-5ca46ceaccd0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh"
Oct 30 01:00:20.289533 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.289365    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw2pm\" (UniqueName: \"kubernetes.io/projected/fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9-kube-api-access-tw2pm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:20.289533 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.289391    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:20.289533 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.289406    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23eea04-ac89-4fa4-bb29-f38d655df0d7-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:20.289533 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.289420    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzhk4\" (UniqueName: \"kubernetes.io/projected/e23eea04-ac89-4fa4-bb29-f38d655df0d7-kube-api-access-mzhk4\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:20.290850 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.290821    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65fe9203-ea37-4a79-9990-5ca46ceaccd0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh\" (UID: \"65fe9203-ea37-4a79-9990-5ca46ceaccd0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh"
Oct 30 01:00:20.313688 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.313665    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk5zj\" (UniqueName: \"kubernetes.io/projected/65fe9203-ea37-4a79-9990-5ca46ceaccd0-kube-api-access-fk5zj\") pod \"payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh\" (UID: \"65fe9203-ea37-4a79-9990-5ca46ceaccd0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh"
Oct 30 01:00:20.388666 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.388639    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh"
Oct 30 01:00:20.470530 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.470485    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl"]
Oct 30 01:00:20.544042 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.544007    6099 generic.go:356] "Generic (PLEG): container finished" podID="ed2d10ae-6748-461e-92c4-f5efc2f0c880" containerID="8b34e01f0fb242100249fbf916dfd8131ad03b5d748e19f23ef791bc5d8c2d57" exitCode=60
Oct 30 01:00:20.544402 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.544144    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc" event={"ID":"ed2d10ae-6748-461e-92c4-f5efc2f0c880","Type":"ContainerDied","Data":"8b34e01f0fb242100249fbf916dfd8131ad03b5d748e19f23ef791bc5d8c2d57"}
Oct 30 01:00:20.544402 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.544201    6099 scope.go:122] "RemoveContainer" containerID="4a1dd6cb9cf50a6d1c5442604282d5504e508652d61ad1f2a648c2ed93add066"
Oct 30 01:00:20.544775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.544750    6099 scope.go:122] "RemoveContainer" containerID="8b34e01f0fb242100249fbf916dfd8131ad03b5d748e19f23ef791bc5d8c2d57"
Oct 30 01:00:20.545235 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:20.545210    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc_odh-ai-gateway-infra(ed2d10ae-6748-461e-92c4-f5efc2f0c880)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc" podUID="ed2d10ae-6748-461e-92c4-f5efc2f0c880"
Oct 30 01:00:20.550746 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.550728    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2"
Oct 30 01:00:20.550853 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.550769    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2" event={"ID":"fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9","Type":"ContainerDied","Data":"0064b32ad697c8721ee878aac548eac71a4ea807b6c41b3064d955054a8a9ac4"}
Oct 30 01:00:20.559004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.558970    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7" event={"ID":"09a29068-f886-4f57-9eab-db7680a24e7a","Type":"ContainerStarted","Data":"00dbd032e17c6207449ccb8df492586b8ce2d068bd1e8fcc99b66edd0fbfa81f"}
Oct 30 01:00:20.560630 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.560602    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl" event={"ID":"2e720881-a65b-4e3b-95c1-e88b04a6c88a","Type":"ContainerStarted","Data":"6d87accc6cf5317b782681b765f90a8cfbc363b129bb514585914f8d417c6cc1"}
Oct 30 01:00:20.563210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.563190    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k"
Oct 30 01:00:20.563358 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.563227    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k" event={"ID":"e23eea04-ac89-4fa4-bb29-f38d655df0d7","Type":"ContainerDied","Data":"172a1f527f7cdcc510552d66fb56c811d47f82c4508bad40f7ad1aed46503ced"}
Oct 30 01:00:20.569518 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.569444    6099 scope.go:122] "RemoveContainer" containerID="e579730b16a3ed3e339bfc46538fd93f4ea94bdd195693819753922f59395fe6"
Oct 30 01:00:20.569655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.569628    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz"]
Oct 30 01:00:20.585450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.584806    6099 scope.go:122] "RemoveContainer" containerID="354ae02f181ea8c23ae98fc0f66e9e27cc718e729da1fa8eb145aaf32cbee8b3"
Oct 30 01:00:20.652409 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.652381    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2"]
Oct 30 01:00:20.678648 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.658098    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-62f5d1-29888700-9pxw2"]
Oct 30 01:00:20.707644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.707612    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh"]
Oct 30 01:00:20.739061 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.738972    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7" podStartSLOduration=20.738954622 podStartE2EDuration="20.738954622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:20.73686755 +0000 UTC m=+1514.593775306" watchObservedRunningTime="2026-10-30 01:00:20.738954622 +0000 UTC m=+1514.595862378"
Oct 30 01:00:20.802680 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.802603    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw"
Oct 30 01:00:20.819647 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.819621    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k"]
Oct 30 01:00:20.828468 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.828434    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ea2547-29888700-bs44k"]
Oct 30 01:00:20.846795 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.846772    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz"
Oct 30 01:00:20.903849 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.903814    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c86d85f-10ed-4300-98d7-e6934451d12d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c86d85f-10ed-4300-98d7-e6934451d12d-maas-api-service-ca\") pod \"9c86d85f-10ed-4300-98d7-e6934451d12d\" (UID: \"9c86d85f-10ed-4300-98d7-e6934451d12d\") "
Oct 30 01:00:20.904140 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.904122    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c86d85f-10ed-4300-98d7-e6934451d12d-kube-api-access-prdh9\" (UniqueName: \"kubernetes.io/projected/9c86d85f-10ed-4300-98d7-e6934451d12d-kube-api-access-prdh9\") pod \"9c86d85f-10ed-4300-98d7-e6934451d12d\" (UID: \"9c86d85f-10ed-4300-98d7-e6934451d12d\") "
Oct 30 01:00:20.904612 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.904283    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c86d85f-10ed-4300-98d7-e6934451d12d-maas-api-service-ca" pod "9c86d85f-10ed-4300-98d7-e6934451d12d" (UID: "9c86d85f-10ed-4300-98d7-e6934451d12d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:20.904879 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.904837    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c86d85f-10ed-4300-98d7-e6934451d12d-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:20.907013 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:20.906982    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c86d85f-10ed-4300-98d7-e6934451d12d-kube-api-access-prdh9" pod "9c86d85f-10ed-4300-98d7-e6934451d12d" (UID: "9c86d85f-10ed-4300-98d7-e6934451d12d"). InnerVolumeSpecName "kube-api-access-prdh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:21.005758 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.005724    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f378df0c-bde8-4326-ade3-7c4ecd956cb3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f378df0c-bde8-4326-ade3-7c4ecd956cb3-maas-api-service-ca\") pod \"f378df0c-bde8-4326-ade3-7c4ecd956cb3\" (UID: \"f378df0c-bde8-4326-ade3-7c4ecd956cb3\") "
Oct 30 01:00:21.005912 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.005842    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f378df0c-bde8-4326-ade3-7c4ecd956cb3-kube-api-access-bm96k\" (UniqueName: \"kubernetes.io/projected/f378df0c-bde8-4326-ade3-7c4ecd956cb3-kube-api-access-bm96k\") pod \"f378df0c-bde8-4326-ade3-7c4ecd956cb3\" (UID: \"f378df0c-bde8-4326-ade3-7c4ecd956cb3\") "
Oct 30 01:00:21.006067 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.006043    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f378df0c-bde8-4326-ade3-7c4ecd956cb3-maas-api-service-ca" pod "f378df0c-bde8-4326-ade3-7c4ecd956cb3" (UID: "f378df0c-bde8-4326-ade3-7c4ecd956cb3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:21.006363 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.006349    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prdh9\" (UniqueName: \"kubernetes.io/projected/9c86d85f-10ed-4300-98d7-e6934451d12d-kube-api-access-prdh9\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:21.006411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.006366    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f378df0c-bde8-4326-ade3-7c4ecd956cb3-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:21.008016 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.007987    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f378df0c-bde8-4326-ade3-7c4ecd956cb3-kube-api-access-bm96k" pod "f378df0c-bde8-4326-ade3-7c4ecd956cb3" (UID: "f378df0c-bde8-4326-ade3-7c4ecd956cb3"). InnerVolumeSpecName "kube-api-access-bm96k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:21.107501 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.107465    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm96k\" (UniqueName: \"kubernetes.io/projected/f378df0c-bde8-4326-ade3-7c4ecd956cb3-kube-api-access-bm96k\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:21.255044 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:21.255012    6099 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/6faaf044bb76fed172811f15938ca1b9eb6f6cb406d7e5f8d57bf8ee9cab22be/diff" to get inode usage: stat /var/lib/containers/storage/overlay/6faaf044bb76fed172811f15938ca1b9eb6f6cb406d7e5f8d57bf8ee9cab22be/diff: no such file or directory, extraDiskErr: <nil>
Oct 30 01:00:21.257151 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:21.257131    6099 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/3a9473d07fbf47bb4623720e09c3215b247b98bf4a62d005a0d4ab3b9581b7f6/diff" to get inode usage: stat /var/lib/containers/storage/overlay/3a9473d07fbf47bb4623720e09c3215b247b98bf4a62d005a0d4ab3b9581b7f6/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/openshift-ingress_payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq_4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a/payload-pre-processing/0.log" to get inode usage: stat /var/log/pods/openshift-ingress_payload-pre-processing-e2e-api-b-601aea-6587946c4c-gzxrq_4f9d8700-79dd-4e1d-8d46-897d4e3a8a1a/payload-pre-processing/0.log: no such file or directory
Oct 30 01:00:21.584039 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.583997    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz"
Oct 30 01:00:21.584500 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.583993    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz" event={"ID":"f378df0c-bde8-4326-ade3-7c4ecd956cb3","Type":"ContainerDied","Data":"10c90a027f14e4d2c658fc9577c0bb6a5ab2b72598fbcd6c3073453da8e7b434"}
Oct 30 01:00:21.584500 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.584145    6099 scope.go:122] "RemoveContainer" containerID="b664363ee8b35afbadfed7c112f3aae8dccfeb532af138ec75bbd148a9d790f1"
Oct 30 01:00:21.586292 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.586259    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh" event={"ID":"65fe9203-ea37-4a79-9990-5ca46ceaccd0","Type":"ContainerStarted","Data":"0852f5fefd82a50ce5cfdda9ad50ad2d6cedce1091c2cfc931cf02ea74707402"}
Oct 30 01:00:21.586482 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.586336    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh" event={"ID":"65fe9203-ea37-4a79-9990-5ca46ceaccd0","Type":"ContainerStarted","Data":"67ac7603bff864aad86429595e5249d08e7b35684b2976d8a7fb01bcb3195239"}
Oct 30 01:00:21.586482 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.586370    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh"
Oct 30 01:00:21.588944 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.588918    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw"
Oct 30 01:00:21.589044 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.588942    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw" event={"ID":"9c86d85f-10ed-4300-98d7-e6934451d12d","Type":"ContainerDied","Data":"e4e6ed088f5ec9dc2ecb687189b416b871ce8dc9b16d0fa23478ef614ab88e7e"}
Oct 30 01:00:21.591346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.591322    6099 generic.go:356] "Generic (PLEG): container finished" podID="09a29068-f886-4f57-9eab-db7680a24e7a" containerID="00dbd032e17c6207449ccb8df492586b8ce2d068bd1e8fcc99b66edd0fbfa81f" exitCode=60
Oct 30 01:00:21.591624 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.591392    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7" event={"ID":"09a29068-f886-4f57-9eab-db7680a24e7a","Type":"ContainerDied","Data":"00dbd032e17c6207449ccb8df492586b8ce2d068bd1e8fcc99b66edd0fbfa81f"}
Oct 30 01:00:21.592241 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.592187    6099 scope.go:122] "RemoveContainer" containerID="00dbd032e17c6207449ccb8df492586b8ce2d068bd1e8fcc99b66edd0fbfa81f"
Oct 30 01:00:21.592597 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:21.592573    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7_odh-ai-gateway-infra(09a29068-f886-4f57-9eab-db7680a24e7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7" podUID="09a29068-f886-4f57-9eab-db7680a24e7a"
Oct 30 01:00:21.594333 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.594310    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl" event={"ID":"2e720881-a65b-4e3b-95c1-e88b04a6c88a","Type":"ContainerStarted","Data":"d9ca08859ff4458412d7ef441e725c575c882d2654f56999ba0d21de7e9687de"}
Oct 30 01:00:21.594427 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.594377    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl"
Oct 30 01:00:21.598594 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.598570    6099 scope.go:122] "RemoveContainer" containerID="e36a0668d7673488b166dd7dbb2bd4459d3d38c4d7ebf9ec5fed238bad91c211"
Oct 30 01:00:21.599539 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.599454    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc"]
Oct 30 01:00:21.608989 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.608970    6099 scope.go:122] "RemoveContainer" containerID="0769681754cb16ec2e57cdc67971934829ee36edf542e457e882443a07a98b94"
Oct 30 01:00:21.644833 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.644802    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw"]
Oct 30 01:00:21.648434 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.648405    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a32412-29888700-xdjcw"]
Oct 30 01:00:21.752639 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.752605    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz"]
Oct 30 01:00:21.758227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.758201    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888700-xkpjz"]
Oct 30 01:00:21.763569 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.763540    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7"]
Oct 30 01:00:21.801271 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.801248    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc"
Oct 30 01:00:21.830262 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.830213    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl" podStartSLOduration=2.830196422 podStartE2EDuration="2.830196422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:21.825361398 +0000 UTC m=+1515.682269153" watchObservedRunningTime="2026-10-30 01:00:21.830196422 +0000 UTC m=+1515.687104177"
Oct 30 01:00:21.859574 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:21.859515    6099 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/8bda00e189ac0d3dad28aff722c0463ac2c2a9ec71e033909f6f7bb944c02bed/diff" to get inode usage: stat /var/lib/containers/storage/overlay/8bda00e189ac0d3dad28aff722c0463ac2c2a9ec71e033909f6f7bb944c02bed/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/openshift-ingress_payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg_4f260864-36c4-4a59-b3a9-b47332908825/payload-processing/0.log" to get inode usage: stat /var/log/pods/openshift-ingress_payload-processing-e2e-api-b-601aea-68cd56fdcb-99xxg_4f260864-36c4-4a59-b3a9-b47332908825/payload-processing/0.log: no such file or directory
Oct 30 01:00:21.867627 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.867561    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh" podStartSLOduration=2.867548561 podStartE2EDuration="2.867548561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:21.864295789 +0000 UTC m=+1515.721203546" watchObservedRunningTime="2026-10-30 01:00:21.867548561 +0000 UTC m=+1515.724456316"
Oct 30 01:00:21.949379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.949341    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed2d10ae-6748-461e-92c4-f5efc2f0c880-kube-api-access-hfk5t\" (UniqueName: \"kubernetes.io/projected/ed2d10ae-6748-461e-92c4-f5efc2f0c880-kube-api-access-hfk5t\") pod \"ed2d10ae-6748-461e-92c4-f5efc2f0c880\" (UID: \"ed2d10ae-6748-461e-92c4-f5efc2f0c880\") "
Oct 30 01:00:21.949547 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.949531    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed2d10ae-6748-461e-92c4-f5efc2f0c880-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d10ae-6748-461e-92c4-f5efc2f0c880-maas-api-service-ca\") pod \"ed2d10ae-6748-461e-92c4-f5efc2f0c880\" (UID: \"ed2d10ae-6748-461e-92c4-f5efc2f0c880\") "
Oct 30 01:00:21.949988 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.949957    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2d10ae-6748-461e-92c4-f5efc2f0c880-maas-api-service-ca" pod "ed2d10ae-6748-461e-92c4-f5efc2f0c880" (UID: "ed2d10ae-6748-461e-92c4-f5efc2f0c880"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:21.950379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.950356    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d10ae-6748-461e-92c4-f5efc2f0c880-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:21.951672 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:21.951645    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2d10ae-6748-461e-92c4-f5efc2f0c880-kube-api-access-hfk5t" pod "ed2d10ae-6748-461e-92c4-f5efc2f0c880" (UID: "ed2d10ae-6748-461e-92c4-f5efc2f0c880"). InnerVolumeSpecName "kube-api-access-hfk5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:22.051504 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:22.051470    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfk5t\" (UniqueName: \"kubernetes.io/projected/ed2d10ae-6748-461e-92c4-f5efc2f0c880-kube-api-access-hfk5t\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:22.265087 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:22.264997    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c86d85f-10ed-4300-98d7-e6934451d12d" path="/var/lib/kubelet/pods/9c86d85f-10ed-4300-98d7-e6934451d12d/volumes"
Oct 30 01:00:22.265543 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:22.265520    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e23eea04-ac89-4fa4-bb29-f38d655df0d7" path="/var/lib/kubelet/pods/e23eea04-ac89-4fa4-bb29-f38d655df0d7/volumes"
Oct 30 01:00:22.266047 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:22.266024    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f378df0c-bde8-4326-ade3-7c4ecd956cb3" path="/var/lib/kubelet/pods/f378df0c-bde8-4326-ade3-7c4ecd956cb3/volumes"
Oct 30 01:00:22.266816 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:22.266796    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9" path="/var/lib/kubelet/pods/fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9/volumes"
Oct 30 01:00:22.605751 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:22.605723    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc"
Oct 30 01:00:22.606203 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:22.605758    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc" event={"ID":"ed2d10ae-6748-461e-92c4-f5efc2f0c880","Type":"ContainerDied","Data":"7ec6e1bd037a065dcc5827dc82180163ad74b021b64cf82f9d57030c50af97fd"}
Oct 30 01:00:22.606203 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:22.605826    6099 scope.go:122] "RemoveContainer" containerID="8b34e01f0fb242100249fbf916dfd8131ad03b5d748e19f23ef791bc5d8c2d57"
Oct 30 01:00:22.697697 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:22.697666    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc"]
Oct 30 01:00:22.707574 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:22.707538    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-40ac41-29888700-mlrxc"]
Oct 30 01:00:22.795676 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:22.795651    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7"
Oct 30 01:00:22.965716 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:22.965612    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a29068-f886-4f57-9eab-db7680a24e7a-kube-api-access-7ntrs\" (UniqueName: \"kubernetes.io/projected/09a29068-f886-4f57-9eab-db7680a24e7a-kube-api-access-7ntrs\") pod \"09a29068-f886-4f57-9eab-db7680a24e7a\" (UID: \"09a29068-f886-4f57-9eab-db7680a24e7a\") "
Oct 30 01:00:22.965862 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:22.965777    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a29068-f886-4f57-9eab-db7680a24e7a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a29068-f886-4f57-9eab-db7680a24e7a-maas-api-service-ca\") pod \"09a29068-f886-4f57-9eab-db7680a24e7a\" (UID: \"09a29068-f886-4f57-9eab-db7680a24e7a\") "
Oct 30 01:00:22.966125 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:22.966102    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a29068-f886-4f57-9eab-db7680a24e7a-maas-api-service-ca" pod "09a29068-f886-4f57-9eab-db7680a24e7a" (UID: "09a29068-f886-4f57-9eab-db7680a24e7a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:22.966381 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:22.966367    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a29068-f886-4f57-9eab-db7680a24e7a-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:22.967894 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:22.967871    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a29068-f886-4f57-9eab-db7680a24e7a-kube-api-access-7ntrs" pod "09a29068-f886-4f57-9eab-db7680a24e7a" (UID: "09a29068-f886-4f57-9eab-db7680a24e7a"). InnerVolumeSpecName "kube-api-access-7ntrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:23.067720 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:23.067685    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ntrs\" (UniqueName: \"kubernetes.io/projected/09a29068-f886-4f57-9eab-db7680a24e7a-kube-api-access-7ntrs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:23.621256 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:23.621225    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7"
Oct 30 01:00:23.621256 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:23.621240    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7" event={"ID":"09a29068-f886-4f57-9eab-db7680a24e7a","Type":"ContainerDied","Data":"57dc81b6fce7b754892dea3aa94e5bf9b2ff5bfbb8e4da6c474067fec287f0e9"}
Oct 30 01:00:23.621783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:23.621332    6099 scope.go:122] "RemoveContainer" containerID="00dbd032e17c6207449ccb8df492586b8ce2d068bd1e8fcc99b66edd0fbfa81f"
Oct 30 01:00:23.681524 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:23.681495    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7"]
Oct 30 01:00:23.695599 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:23.695564    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888700-xnbt7"]
Oct 30 01:00:24.263640 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:24.263605    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a29068-f886-4f57-9eab-db7680a24e7a" path="/var/lib/kubelet/pods/09a29068-f886-4f57-9eab-db7680a24e7a/volumes"
Oct 30 01:00:24.264182 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:24.264165    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed2d10ae-6748-461e-92c4-f5efc2f0c880" path="/var/lib/kubelet/pods/ed2d10ae-6748-461e-92c4-f5efc2f0c880/volumes"
Oct 30 01:00:27.967895 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:27.967862    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:00:31.241294 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.241250    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"]
Oct 30 01:00:31.242202 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242176    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e23eea04-ac89-4fa4-bb29-f38d655df0d7" containerName="cleanup"
Oct 30 01:00:31.242202 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242203    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23eea04-ac89-4fa4-bb29-f38d655df0d7" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242234    6099 cpu_manager.go:414] "containerMap: removing container" podUID="ed2d10ae-6748-461e-92c4-f5efc2f0c880" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242243    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed2d10ae-6748-461e-92c4-f5efc2f0c880" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242253    6099 cpu_manager.go:414] "containerMap: removing container" podUID="fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242261    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242277    6099 cpu_manager.go:414] "containerMap: removing container" podUID="ed2d10ae-6748-461e-92c4-f5efc2f0c880" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242284    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed2d10ae-6748-461e-92c4-f5efc2f0c880" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242293    6099 cpu_manager.go:414] "containerMap: removing container" podUID="9c86d85f-10ed-4300-98d7-e6934451d12d" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242299    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c86d85f-10ed-4300-98d7-e6934451d12d" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242311    6099 cpu_manager.go:414] "containerMap: removing container" podUID="9c86d85f-10ed-4300-98d7-e6934451d12d" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242317    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c86d85f-10ed-4300-98d7-e6934451d12d" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242342    6099 cpu_manager.go:414] "containerMap: removing container" podUID="fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242348    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242358    6099 cpu_manager.go:414] "containerMap: removing container" podUID="f378df0c-bde8-4326-ade3-7c4ecd956cb3" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242364    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f378df0c-bde8-4326-ade3-7c4ecd956cb3" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242375    6099 cpu_manager.go:414] "containerMap: removing container" podUID="9c86d85f-10ed-4300-98d7-e6934451d12d" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242382    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c86d85f-10ed-4300-98d7-e6934451d12d" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242390    6099 cpu_manager.go:414] "containerMap: removing container" podUID="09a29068-f886-4f57-9eab-db7680a24e7a" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242396    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a29068-f886-4f57-9eab-db7680a24e7a" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242411    6099 cpu_manager.go:414] "containerMap: removing container" podUID="f378df0c-bde8-4326-ade3-7c4ecd956cb3" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242418    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f378df0c-bde8-4326-ade3-7c4ecd956cb3" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242435    6099 cpu_manager.go:414] "containerMap: removing container" podUID="09a29068-f886-4f57-9eab-db7680a24e7a" containerName="cleanup"
Oct 30 01:00:31.242445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242441    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a29068-f886-4f57-9eab-db7680a24e7a" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242466    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e23eea04-ac89-4fa4-bb29-f38d655df0d7" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242473    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23eea04-ac89-4fa4-bb29-f38d655df0d7" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242492    6099 cpu_manager.go:414] "containerMap: removing container" podUID="f378df0c-bde8-4326-ade3-7c4ecd956cb3" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242497    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f378df0c-bde8-4326-ade3-7c4ecd956cb3" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242511    6099 cpu_manager.go:414] "containerMap: removing container" podUID="09a29068-f886-4f57-9eab-db7680a24e7a" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242517    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a29068-f886-4f57-9eab-db7680a24e7a" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242640    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242649    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23eea04-ac89-4fa4-bb29-f38d655df0d7" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242657    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed2d10ae-6748-461e-92c4-f5efc2f0c880" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242662    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f378df0c-bde8-4326-ade3-7c4ecd956cb3" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242668    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a29068-f886-4f57-9eab-db7680a24e7a" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242674    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23eea04-ac89-4fa4-bb29-f38d655df0d7" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242681    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed2d10ae-6748-461e-92c4-f5efc2f0c880" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242687    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242692    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed2d10ae-6748-461e-92c4-f5efc2f0c880" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242697    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c86d85f-10ed-4300-98d7-e6934451d12d" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242718    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c86d85f-10ed-4300-98d7-e6934451d12d" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242730    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242740    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f378df0c-bde8-4326-ade3-7c4ecd956cb3" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242748    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c86d85f-10ed-4300-98d7-e6934451d12d" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242755    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f378df0c-bde8-4326-ade3-7c4ecd956cb3" containerName="cleanup"
Oct 30 01:00:31.243121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.242763    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a29068-f886-4f57-9eab-db7680a24e7a" containerName="cleanup"
Oct 30 01:00:31.246385 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.246365    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.261536 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.261508    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-820c04-openshift-default-dockercfg-rnmpt\""
Oct 30 01:00:31.277039 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.277004    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"]
Oct 30 01:00:31.344341 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.344301    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-data\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.344589 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.344562    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-workload-certs\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.344659 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.344606    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istiod-ca-cert\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.344916 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.344893    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-envoy\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.345003 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.344982    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-podinfo\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.345118 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.345047    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-token\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.345175 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.345136    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-credential-socket\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.345376 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.345177    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nctt7\" (UniqueName: \"kubernetes.io/projected/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-kube-api-access-nctt7\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.345376 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.345256    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-workload-socket\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.451729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.447112    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-envoy\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.451729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.447167    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-podinfo\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.451729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.447224    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-token\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.451729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.447275    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-credential-socket\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.451729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.447309    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nctt7\" (UniqueName: \"kubernetes.io/projected/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-kube-api-access-nctt7\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.451729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.447361    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-workload-socket\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.451729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.447414    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-data\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.451729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.447586    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-workload-certs\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.451729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.447607    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istiod-ca-cert\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.451729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.448593    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istiod-ca-cert\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.451729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.450138    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-workload-certs\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.451729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.450402    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-workload-socket\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.451729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.450681    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-data\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.451729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.450951    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-credential-socket\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.489149 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.481661    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-envoy\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.494228 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.490477    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-token\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.494228 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.494114    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-podinfo\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.513738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.508459    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nctt7\" (UniqueName: \"kubernetes.io/projected/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-kube-api-access-nctt7\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.565042 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.564692    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:31.893749 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.893723    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"]
Oct 30 01:00:31.895605 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.895559    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:31.895727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.895632    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:31.895727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:31.895667    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:32.614191 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:32.614162    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl"
Oct 30 01:00:32.614595 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:32.614246    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh"
Oct 30 01:00:32.716873 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:32.716838    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44" event={"ID":"7c6446b4-b8c2-42b3-b449-2ae9141a19dc","Type":"ContainerStarted","Data":"734491f5f7fe82ea06f75bf7eee81e6b6d172c3a1961436b460e56a17d464e0b"}
Oct 30 01:00:32.717113 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:32.716891    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44" event={"ID":"7c6446b4-b8c2-42b3-b449-2ae9141a19dc","Type":"ContainerStarted","Data":"7299ee14f309a39a0e2382bbee4b2ea2436d590088992aa34bea72b4ed263768"}
Oct 30 01:00:32.780418 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:32.780368    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44" podStartSLOduration=1.780351995 podStartE2EDuration="1.780351995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:32.77802423 +0000 UTC m=+1526.634931987" watchObservedRunningTime="2026-10-30 01:00:32.780351995 +0000 UTC m=+1526.637259814"
Oct 30 01:00:33.565175 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:33.565138    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:33.571621 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:33.571594    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:33.724508 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:33.724477    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:33.725543 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:33.725521    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:00:34.284977 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.284945    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"]
Oct 30 01:00:34.285995 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.285969    6099 cpu_manager.go:414] "containerMap: removing container" podUID="ed2d10ae-6748-461e-92c4-f5efc2f0c880" containerName="cleanup"
Oct 30 01:00:34.285995 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.285991    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed2d10ae-6748-461e-92c4-f5efc2f0c880" containerName="cleanup"
Oct 30 01:00:34.286176 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.286015    6099 cpu_manager.go:414] "containerMap: removing container" podUID="fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9" containerName="cleanup"
Oct 30 01:00:34.286176 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.286021    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd1e0fff-0f9f-4dfa-89e8-c8b28ae9cbc9" containerName="cleanup"
Oct 30 01:00:34.286176 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.286064    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e23eea04-ac89-4fa4-bb29-f38d655df0d7" containerName="cleanup"
Oct 30 01:00:34.286176 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.286070    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23eea04-ac89-4fa4-bb29-f38d655df0d7" containerName="cleanup"
Oct 30 01:00:34.286292 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.286189    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a29068-f886-4f57-9eab-db7680a24e7a" containerName="cleanup"
Oct 30 01:00:34.286292 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.286199    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23eea04-ac89-4fa4-bb29-f38d655df0d7" containerName="cleanup"
Oct 30 01:00:34.291164 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.291145    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.326335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.326301    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"]
Oct 30 01:00:34.358364 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.358334    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a07920-openshift-default-dockercfg-pc9lh\""
Oct 30 01:00:34.408785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.403937    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-data\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.408785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.403989    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlrtb\" (UniqueName: \"kubernetes.io/projected/f85bad1f-83ed-42b6-b88a-c1388b5480b2-kube-api-access-hlrtb\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.408785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.404151    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-podinfo\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.408785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.404277    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-workload-certs\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.408785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.404332    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-workload-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.408785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.404472    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istiod-ca-cert\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.408785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.404596    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-credential-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.408785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.404624    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-envoy\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.408785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.404778    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-token\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.511729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.507730    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-data\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.511729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.507786    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlrtb\" (UniqueName: \"kubernetes.io/projected/f85bad1f-83ed-42b6-b88a-c1388b5480b2-kube-api-access-hlrtb\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.511729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.507838    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-podinfo\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.511729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.507907    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-workload-certs\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.511729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.507943    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-workload-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.511729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.508076    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istiod-ca-cert\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.511729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.508180    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-credential-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.511729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.508208    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-envoy\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.511729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.508312    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-token\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.511729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.509208    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-data\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.511729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.510249    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istiod-ca-cert\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.511729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.510409    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-workload-certs\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.511729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.510557    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-workload-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.511729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.510727    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-credential-socket\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.522759 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.513387    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-podinfo\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.530393 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.528011    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-envoy\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.544598 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.544563    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlrtb\" (UniqueName: \"kubernetes.io/projected/f85bad1f-83ed-42b6-b88a-c1388b5480b2-kube-api-access-hlrtb\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.546321 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.546288    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-token\") pod \"e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") " pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.614726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.608697    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:34.960658 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.960623    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"]
Oct 30 01:00:34.965143 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.965107    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:34.965248 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.965180    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:34.965248 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:34.965212    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:35.778774 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:35.778700    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48" event={"ID":"f85bad1f-83ed-42b6-b88a-c1388b5480b2","Type":"ContainerStarted","Data":"a1f83ceeed3b9e9fcedc6c53c3b8dea1ca41a624a2b71466bf8ce695c80c77fa"}
Oct 30 01:00:35.779100 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:35.778805    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48" event={"ID":"f85bad1f-83ed-42b6-b88a-c1388b5480b2","Type":"ContainerStarted","Data":"9bd0b63a06da77093a97d940611d6b571ccf8028bc2ce2061136121d235d0047"}
Oct 30 01:00:35.844667 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:35.844616    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48" podStartSLOduration=1.8446022420000001 podStartE2EDuration="1.844602242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:35.842192012 +0000 UTC m=+1529.699099766" watchObservedRunningTime="2026-10-30 01:00:35.844602242 +0000 UTC m=+1529.701509997"
Oct 30 01:00:36.380629 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:36.380564    6099 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23eea04_ac89_4fa4_bb29_f38d655df0d7.slice/crio-conmon-354ae02f181ea8c23ae98fc0f66e9e27cc718e729da1fa8eb145aaf32cbee8b3.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23eea04_ac89_4fa4_bb29_f38d655df0d7.slice/crio-conmon-354ae02f181ea8c23ae98fc0f66e9e27cc718e729da1fa8eb145aaf32cbee8b3.scope: no such file or directory
Oct 30 01:00:36.381196 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:36.381156    6099 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3365bf0b_a96c_429e_8565_faa44fef8ee0.slice/crio-conmon-a2dffcf9e780f7f25f2a88df53d4a7f24345f719a9936412b915df508d4ef5ef.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3365bf0b_a96c_429e_8565_faa44fef8ee0.slice/crio-conmon-a2dffcf9e780f7f25f2a88df53d4a7f24345f719a9936412b915df508d4ef5ef.scope: no such file or directory
Oct 30 01:00:36.381325 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:36.381306    6099 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2294af_beb7_4d49_8cd8_4abd0c04a40a.slice/crio-conmon-e7dd16bcc4154912f278fba96e65c7ce68d8998a9c0e4e9b8594d32c78f5cced.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2294af_beb7_4d49_8cd8_4abd0c04a40a.slice/crio-conmon-e7dd16bcc4154912f278fba96e65c7ce68d8998a9c0e4e9b8594d32c78f5cced.scope: no such file or directory
Oct 30 01:00:36.381440 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:36.381423    6099 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2294af_beb7_4d49_8cd8_4abd0c04a40a.slice/crio-e7dd16bcc4154912f278fba96e65c7ce68d8998a9c0e4e9b8594d32c78f5cced.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2294af_beb7_4d49_8cd8_4abd0c04a40a.slice/crio-e7dd16bcc4154912f278fba96e65c7ce68d8998a9c0e4e9b8594d32c78f5cced.scope: no such file or directory
Oct 30 01:00:36.381548 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:36.381531    6099 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3365bf0b_a96c_429e_8565_faa44fef8ee0.slice/crio-a2dffcf9e780f7f25f2a88df53d4a7f24345f719a9936412b915df508d4ef5ef.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3365bf0b_a96c_429e_8565_faa44fef8ee0.slice/crio-a2dffcf9e780f7f25f2a88df53d4a7f24345f719a9936412b915df508d4ef5ef.scope: no such file or directory
Oct 30 01:00:36.381673 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:36.381655    6099 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23eea04_ac89_4fa4_bb29_f38d655df0d7.slice/crio-354ae02f181ea8c23ae98fc0f66e9e27cc718e729da1fa8eb145aaf32cbee8b3.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23eea04_ac89_4fa4_bb29_f38d655df0d7.slice/crio-354ae02f181ea8c23ae98fc0f66e9e27cc718e729da1fa8eb145aaf32cbee8b3.scope: no such file or directory
Oct 30 01:00:36.381840 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:36.381819    6099 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c86d85f_10ed_4300_98d7_e6934451d12d.slice/crio-conmon-e36a0668d7673488b166dd7dbb2bd4459d3d38c4d7ebf9ec5fed238bad91c211.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c86d85f_10ed_4300_98d7_e6934451d12d.slice/crio-conmon-e36a0668d7673488b166dd7dbb2bd4459d3d38c4d7ebf9ec5fed238bad91c211.scope: no such file or directory
Oct 30 01:00:36.381991 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:36.381959    6099 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1e0fff_0f9f_4dfa_89e8_c8b28ae9cbc9.slice/crio-conmon-e579730b16a3ed3e339bfc46538fd93f4ea94bdd195693819753922f59395fe6.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1e0fff_0f9f_4dfa_89e8_c8b28ae9cbc9.slice/crio-conmon-e579730b16a3ed3e339bfc46538fd93f4ea94bdd195693819753922f59395fe6.scope: no such file or directory
Oct 30 01:00:36.382112 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:36.382003    6099 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c86d85f_10ed_4300_98d7_e6934451d12d.slice/crio-e36a0668d7673488b166dd7dbb2bd4459d3d38c4d7ebf9ec5fed238bad91c211.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c86d85f_10ed_4300_98d7_e6934451d12d.slice/crio-e36a0668d7673488b166dd7dbb2bd4459d3d38c4d7ebf9ec5fed238bad91c211.scope: no such file or directory
Oct 30 01:00:36.382112 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:36.382028    6099 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1e0fff_0f9f_4dfa_89e8_c8b28ae9cbc9.slice/crio-e579730b16a3ed3e339bfc46538fd93f4ea94bdd195693819753922f59395fe6.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1e0fff_0f9f_4dfa_89e8_c8b28ae9cbc9.slice/crio-e579730b16a3ed3e339bfc46538fd93f4ea94bdd195693819753922f59395fe6.scope: no such file or directory
Oct 30 01:00:36.427728 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:36.416590    6099 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf378df0c_bde8_4326_ade3_7c4ecd956cb3.slice/crio-conmon-b664363ee8b35afbadfed7c112f3aae8dccfeb532af138ec75bbd148a9d790f1.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf378df0c_bde8_4326_ade3_7c4ecd956cb3.slice/crio-conmon-b664363ee8b35afbadfed7c112f3aae8dccfeb532af138ec75bbd148a9d790f1.scope: no such file or directory
Oct 30 01:00:36.427728 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:36.416649    6099 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf378df0c_bde8_4326_ade3_7c4ecd956cb3.slice/crio-b664363ee8b35afbadfed7c112f3aae8dccfeb532af138ec75bbd148a9d790f1.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf378df0c_bde8_4326_ade3_7c4ecd956cb3.slice/crio-b664363ee8b35afbadfed7c112f3aae8dccfeb532af138ec75bbd148a9d790f1.scope: no such file or directory
Oct 30 01:00:36.427728 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:36.416674    6099 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2d10ae_6748_461e_92c4_f5efc2f0c880.slice/crio-conmon-8b34e01f0fb242100249fbf916dfd8131ad03b5d748e19f23ef791bc5d8c2d57.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2d10ae_6748_461e_92c4_f5efc2f0c880.slice/crio-conmon-8b34e01f0fb242100249fbf916dfd8131ad03b5d748e19f23ef791bc5d8c2d57.scope: no such file or directory
Oct 30 01:00:36.427728 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:36.416696    6099 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a29068_f886_4f57_9eab_db7680a24e7a.slice/crio-conmon-00dbd032e17c6207449ccb8df492586b8ce2d068bd1e8fcc99b66edd0fbfa81f.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a29068_f886_4f57_9eab_db7680a24e7a.slice/crio-conmon-00dbd032e17c6207449ccb8df492586b8ce2d068bd1e8fcc99b66edd0fbfa81f.scope: no such file or directory
Oct 30 01:00:36.427728 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:36.416775    6099 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2d10ae_6748_461e_92c4_f5efc2f0c880.slice/crio-8b34e01f0fb242100249fbf916dfd8131ad03b5d748e19f23ef791bc5d8c2d57.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2d10ae_6748_461e_92c4_f5efc2f0c880.slice/crio-8b34e01f0fb242100249fbf916dfd8131ad03b5d748e19f23ef791bc5d8c2d57.scope: no such file or directory
Oct 30 01:00:36.457725 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:36.451874    6099 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f260864_36c4_4a59_b3a9_b47332908825.slice/crio-1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e9d22c_b2b4_4795_be11_ff2d98383a92.slice/crio-8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e190a98_17fe_46f6_adcf_34223eaf820e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e9d22c_b2b4_4795_be11_ff2d98383a92.slice/crio-c3e728f58edd178b4e3c9c0d40c226b9db772799c90b0b478664f719802d0798\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f260864_36c4_4a59_b3a9_b47332908825.slice/crio-conmon-1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20857824_d001_4494_97f3_ff58b868c3c2.slice/crio-6ff55f112fc5499244152c6694cf55212cc4844ab1b85f0cb3b8fe3c6931cf74\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice/crio-7e2678fa240ef4837699a8f6bae1f5cfbaa1664f174092e6001d8be60e30580b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6474c172_c1ba_43d1_a965_242e6b93a3a2.slice/crio-e25508c55536b130cb5c39586f2260eb079617e7dc7bcd3233bd6068ef0d75b6\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e190a98_17fe_46f6_adcf_34223eaf820e.slice/crio-conmon-28a44dc7433d5dd5158e619d7485433c6e041166705909aea99f6cb0bec6ea48.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice/crio-3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f260864_36c4_4a59_b3a9_b47332908825.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20857824_d001_4494_97f3_ff58b868c3c2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20857824_d001_4494_97f3_ff58b868c3c2.slice/crio-21b2b68e62de806c328d1946643254519cab3e76f988257b0e72908f9ab25477.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f260864_36c4_4a59_b3a9_b47332908825.slice/crio-282b21726f130a621a472ee44fd75b56172105a5a97847b7a5cb1803bcbfc181\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e190a98_17fe_46f6_adcf_34223eaf820e.slice/crio-28a44dc7433d5dd5158e619d7485433c6e041166705909aea99f6cb0bec6ea48.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice/crio-conmon-3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6474c172_c1ba_43d1_a965_242e6b93a3a2.slice/crio-conmon-59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20857824_d001_4494_97f3_ff58b868c3c2.slice/crio-conmon-21b2b68e62de806c328d1946643254519cab3e76f988257b0e72908f9ab25477.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6474c172_c1ba_43d1_a965_242e6b93a3a2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6474c172_c1ba_43d1_a965_242e6b93a3a2.slice/crio-59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e9d22c_b2b4_4795_be11_ff2d98383a92.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e9d22c_b2b4_4795_be11_ff2d98383a92.slice/crio-conmon-8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 01:00:36.457725 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:36.452268    6099 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e190a98_17fe_46f6_adcf_34223eaf820e.slice/crio-ac59ecf0ce90b52bd1f24fe98fb3c11961c07b66e0d7d9de08145af46595bdaf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20857824_d001_4494_97f3_ff58b868c3c2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6474c172_c1ba_43d1_a965_242e6b93a3a2.slice/crio-59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e9d22c_b2b4_4795_be11_ff2d98383a92.slice/crio-conmon-8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20857824_d001_4494_97f3_ff58b868c3c2.slice/crio-6ff55f112fc5499244152c6694cf55212cc4844ab1b85f0cb3b8fe3c6931cf74\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20857824_d001_4494_97f3_ff58b868c3c2.slice/crio-conmon-21b2b68e62de806c328d1946643254519cab3e76f988257b0e72908f9ab25477.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20857824_d001_4494_97f3_ff58b868c3c2.slice/crio-21b2b68e62de806c328d1946643254519cab3e76f988257b0e72908f9ab25477.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e9d22c_b2b4_4795_be11_ff2d98383a92.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e190a98_17fe_46f6_adcf_34223eaf820e.slice/crio-28a44dc7433d5dd5158e619d7485433c6e041166705909aea99f6cb0bec6ea48.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f260864_36c4_4a59_b3a9_b47332908825.slice/crio-1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e9d22c_b2b4_4795_be11_ff2d98383a92.slice/crio-8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6474c172_c1ba_43d1_a965_242e6b93a3a2.slice/crio-conmon-59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f260864_36c4_4a59_b3a9_b47332908825.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e190a98_17fe_46f6_adcf_34223eaf820e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice/crio-7e2678fa240ef4837699a8f6bae1f5cfbaa1664f174092e6001d8be60e30580b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f260864_36c4_4a59_b3a9_b47332908825.slice/crio-282b21726f130a621a472ee44fd75b56172105a5a97847b7a5cb1803bcbfc181\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6474c172_c1ba_43d1_a965_242e6b93a3a2.slice/crio-e25508c55536b130cb5c39586f2260eb079617e7dc7bcd3233bd6068ef0d75b6\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice/crio-conmon-3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice/crio-3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 01:00:36.457725 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:36.456585    6099 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f260864_36c4_4a59_b3a9_b47332908825.slice/crio-282b21726f130a621a472ee44fd75b56172105a5a97847b7a5cb1803bcbfc181\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1e0fff_0f9f_4dfa_89e8_c8b28ae9cbc9.slice/crio-0064b32ad697c8721ee878aac548eac71a4ea807b6c41b3064d955054a8a9ac4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c86d85f_10ed_4300_98d7_e6934451d12d.slice/crio-e4e6ed088f5ec9dc2ecb687189b416b871ce8dc9b16d0fa23478ef614ab88e7e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6474c172_c1ba_43d1_a965_242e6b93a3a2.slice/crio-conmon-59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20857824_d001_4494_97f3_ff58b868c3c2.slice/crio-6ff55f112fc5499244152c6694cf55212cc4844ab1b85f0cb3b8fe3c6931cf74\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice/crio-7e2678fa240ef4837699a8f6bae1f5cfbaa1664f174092e6001d8be60e30580b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e9d22c_b2b4_4795_be11_ff2d98383a92.slice/crio-c3e728f58edd178b4e3c9c0d40c226b9db772799c90b0b478664f719802d0798\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6474c172_c1ba_43d1_a965_242e6b93a3a2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20857824_d001_4494_97f3_ff58b868c3c2.slice/crio-21b2b68e62de806c328d1946643254519cab3e76f988257b0e72908f9ab25477.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e190a98_17fe_46f6_adcf_34223eaf820e.slice/crio-ac59ecf0ce90b52bd1f24fe98fb3c11961c07b66e0d7d9de08145af46595bdaf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f260864_36c4_4a59_b3a9_b47332908825.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e9d22c_b2b4_4795_be11_ff2d98383a92.slice/crio-8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 01:00:36.458872 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:36.458796    6099 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice/crio-7e2678fa240ef4837699a8f6bae1f5cfbaa1664f174092e6001d8be60e30580b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f260864_36c4_4a59_b3a9_b47332908825.slice/crio-conmon-1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f260864_36c4_4a59_b3a9_b47332908825.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f260864_36c4_4a59_b3a9_b47332908825.slice/crio-1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e190a98_17fe_46f6_adcf_34223eaf820e.slice/crio-28a44dc7433d5dd5158e619d7485433c6e041166705909aea99f6cb0bec6ea48.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e190a98_17fe_46f6_adcf_34223eaf820e.slice/crio-ac59ecf0ce90b52bd1f24fe98fb3c11961c07b66e0d7d9de08145af46595bdaf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6474c172_c1ba_43d1_a965_242e6b93a3a2.slice/crio-e25508c55536b130cb5c39586f2260eb079617e7dc7bcd3233bd6068ef0d75b6\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6474c172_c1ba_43d1_a965_242e6b93a3a2.slice/crio-conmon-59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice/crio-conmon-3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice/crio-3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20857824_d001_4494_97f3_ff58b868c3c2.slice/crio-conmon-21b2b68e62de806c328d1946643254519cab3e76f988257b0e72908f9ab25477.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6474c172_c1ba_43d1_a965_242e6b93a3a2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e9d22c_b2b4_4795_be11_ff2d98383a92.slice/crio-c3e728f58edd178b4e3c9c0d40c226b9db772799c90b0b478664f719802d0798\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20857824_d001_4494_97f3_ff58b868c3c2.slice/crio-21b2b68e62de806c328d1946643254519cab3e76f988257b0e72908f9ab25477.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 01:00:36.462586 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:36.457352    6099 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a29068_f886_4f57_9eab_db7680a24e7a.slice/crio-00dbd032e17c6207449ccb8df492586b8ce2d068bd1e8fcc99b66edd0fbfa81f.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a29068_f886_4f57_9eab_db7680a24e7a.slice/crio-00dbd032e17c6207449ccb8df492586b8ce2d068bd1e8fcc99b66edd0fbfa81f.scope: no such file or directory
Oct 30 01:00:36.470618 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:36.470504    6099 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f260864_36c4_4a59_b3a9_b47332908825.slice/crio-1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20857824_d001_4494_97f3_ff58b868c3c2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6474c172_c1ba_43d1_a965_242e6b93a3a2.slice/crio-conmon-59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6474c172_c1ba_43d1_a965_242e6b93a3a2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6474c172_c1ba_43d1_a965_242e6b93a3a2.slice/crio-59e5b7a9066560723cdd638673c5c6f9032ffc6a18494d428e4766c726854764.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20857824_d001_4494_97f3_ff58b868c3c2.slice/crio-21b2b68e62de806c328d1946643254519cab3e76f988257b0e72908f9ab25477.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20857824_d001_4494_97f3_ff58b868c3c2.slice/crio-conmon-21b2b68e62de806c328d1946643254519cab3e76f988257b0e72908f9ab25477.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e190a98_17fe_46f6_adcf_34223eaf820e.slice/crio-conmon-28a44dc7433d5dd5158e619d7485433c6e041166705909aea99f6cb0bec6ea48.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e9d22c_b2b4_4795_be11_ff2d98383a92.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e9d22c_b2b4_4795_be11_ff2d98383a92.slice/crio-conmon-8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20857824_d001_4494_97f3_ff58b868c3c2.slice/crio-6ff55f112fc5499244152c6694cf55212cc4844ab1b85f0cb3b8fe3c6931cf74\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f260864_36c4_4a59_b3a9_b47332908825.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e190a98_17fe_46f6_adcf_34223eaf820e.slice/crio-ac59ecf0ce90b52bd1f24fe98fb3c11961c07b66e0d7d9de08145af46595bdaf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice/crio-7e2678fa240ef4837699a8f6bae1f5cfbaa1664f174092e6001d8be60e30580b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e9d22c_b2b4_4795_be11_ff2d98383a92.slice/crio-c3e728f58edd178b4e3c9c0d40c226b9db772799c90b0b478664f719802d0798\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3365bf0b_a96c_429e_8565_faa44fef8ee0.slice/crio-2cf80dc01dcee76806582442e72d51c4a102ec44f94a03eb64e9df174f648544\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2294af_beb7_4d49_8cd8_4abd0c04a40a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e190a98_17fe_46f6_adcf_34223eaf820e.slice/crio-28a44dc7433d5dd5158e619d7485433c6e041166705909aea99f6cb0bec6ea48.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e190a98_17fe_46f6_adcf_34223eaf820e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f260864_36c4_4a59_b3a9_b47332908825.slice/crio-282b21726f130a621a472ee44fd75b56172105a5a97847b7a5cb1803bcbfc181\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice/crio-conmon-3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f260864_36c4_4a59_b3a9_b47332908825.slice/crio-conmon-1abe11ecad261a8c6e85c5ef02d79244df89866d381fc07f6c9d89920e8b59af.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d8700_79dd_4e1d_8d46_897d4e3a8a1a.slice/crio-3e5624455dc65c9aa54c044caacaa31f0c99aeba042abf787f60a2f964a8c3e5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e9d22c_b2b4_4795_be11_ff2d98383a92.slice/crio-8a631e2ffdb7fdad1c9f941228750c57ddc8ad704b30bc67cb7f55dc3f457853.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 01:00:36.611045 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.610961    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:36.622617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.621698    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:36.682030 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.681999    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw"
Oct 30 01:00:36.791814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.791778    6099 generic.go:356] "Generic (PLEG): container finished" podID="020ef144-858f-4125-a9b8-f18fbebbffad" containerID="a331f8481dc3d340991fa433417ca5d071e346625bd5e76932ede083f43e4da9" exitCode=137
Oct 30 01:00:36.792000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.791839    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw"
Oct 30 01:00:36.792000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.791858    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw" event={"ID":"020ef144-858f-4125-a9b8-f18fbebbffad","Type":"ContainerDied","Data":"a331f8481dc3d340991fa433417ca5d071e346625bd5e76932ede083f43e4da9"}
Oct 30 01:00:36.792000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.791946    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw" event={"ID":"020ef144-858f-4125-a9b8-f18fbebbffad","Type":"ContainerDied","Data":"09bcb6a81bb59fc4ead064544b7458082e2905ed315b14d0739140772d5e7d25"}
Oct 30 01:00:36.792000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.791975    6099 scope.go:122] "RemoveContainer" containerID="a331f8481dc3d340991fa433417ca5d071e346625bd5e76932ede083f43e4da9"
Oct 30 01:00:36.792563 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.792475    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:36.793885 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.793617    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:00:36.808671 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.808647    6099 scope.go:122] "RemoveContainer" containerID="44d3f0bc3f3bacb719855b5c4621dec4588bb9ca0d1068987c8c16be4a634efa"
Oct 30 01:00:36.821459 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.818179    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/020ef144-858f-4125-a9b8-f18fbebbffad-kube-api-access-cq9cr\" (UniqueName: \"kubernetes.io/projected/020ef144-858f-4125-a9b8-f18fbebbffad-kube-api-access-cq9cr\") pod \"020ef144-858f-4125-a9b8-f18fbebbffad\" (UID: \"020ef144-858f-4125-a9b8-f18fbebbffad\") "
Oct 30 01:00:36.821459 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.818961    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/020ef144-858f-4125-a9b8-f18fbebbffad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020ef144-858f-4125-a9b8-f18fbebbffad-maas-api-service-ca\") pod \"020ef144-858f-4125-a9b8-f18fbebbffad\" (UID: \"020ef144-858f-4125-a9b8-f18fbebbffad\") "
Oct 30 01:00:36.821459 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.820069    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020ef144-858f-4125-a9b8-f18fbebbffad-maas-api-service-ca" pod "020ef144-858f-4125-a9b8-f18fbebbffad" (UID: "020ef144-858f-4125-a9b8-f18fbebbffad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:36.825782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.825026    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020ef144-858f-4125-a9b8-f18fbebbffad-kube-api-access-cq9cr" pod "020ef144-858f-4125-a9b8-f18fbebbffad" (UID: "020ef144-858f-4125-a9b8-f18fbebbffad"). InnerVolumeSpecName "kube-api-access-cq9cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:36.826862 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.826847    6099 scope.go:122] "RemoveContainer" containerID="a331f8481dc3d340991fa433417ca5d071e346625bd5e76932ede083f43e4da9"
Oct 30 01:00:36.827173 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:36.827149    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a331f8481dc3d340991fa433417ca5d071e346625bd5e76932ede083f43e4da9\": container with ID starting with a331f8481dc3d340991fa433417ca5d071e346625bd5e76932ede083f43e4da9 not found: ID does not exist" containerID="a331f8481dc3d340991fa433417ca5d071e346625bd5e76932ede083f43e4da9"
Oct 30 01:00:36.827262 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.827187    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a331f8481dc3d340991fa433417ca5d071e346625bd5e76932ede083f43e4da9"} err="failed to get container status \"a331f8481dc3d340991fa433417ca5d071e346625bd5e76932ede083f43e4da9\": rpc error: code = NotFound desc = could not find container \"a331f8481dc3d340991fa433417ca5d071e346625bd5e76932ede083f43e4da9\": container with ID starting with a331f8481dc3d340991fa433417ca5d071e346625bd5e76932ede083f43e4da9 not found: ID does not exist"
Oct 30 01:00:36.827262 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.827211    6099 scope.go:122] "RemoveContainer" containerID="44d3f0bc3f3bacb719855b5c4621dec4588bb9ca0d1068987c8c16be4a634efa"
Oct 30 01:00:36.827524 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:36.827499    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d3f0bc3f3bacb719855b5c4621dec4588bb9ca0d1068987c8c16be4a634efa\": container with ID starting with 44d3f0bc3f3bacb719855b5c4621dec4588bb9ca0d1068987c8c16be4a634efa not found: ID does not exist" containerID="44d3f0bc3f3bacb719855b5c4621dec4588bb9ca0d1068987c8c16be4a634efa"
Oct 30 01:00:36.827614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.827529    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d3f0bc3f3bacb719855b5c4621dec4588bb9ca0d1068987c8c16be4a634efa"} err="failed to get container status \"44d3f0bc3f3bacb719855b5c4621dec4588bb9ca0d1068987c8c16be4a634efa\": rpc error: code = NotFound desc = could not find container \"44d3f0bc3f3bacb719855b5c4621dec4588bb9ca0d1068987c8c16be4a634efa\": container with ID starting with 44d3f0bc3f3bacb719855b5c4621dec4588bb9ca0d1068987c8c16be4a634efa not found: ID does not exist"
Oct 30 01:00:36.921421 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.921332    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020ef144-858f-4125-a9b8-f18fbebbffad-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:36.921421 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:36.921367    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq9cr\" (UniqueName: \"kubernetes.io/projected/020ef144-858f-4125-a9b8-f18fbebbffad-kube-api-access-cq9cr\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:37.143310 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.143277    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw"]
Oct 30 01:00:37.148101 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.148070    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73b32e-29888700-xl9sw"]
Oct 30 01:00:37.524403 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.524365    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd"]
Oct 30 01:00:37.525335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.525311    6099 cpu_manager.go:414] "containerMap: removing container" podUID="020ef144-858f-4125-a9b8-f18fbebbffad" containerName="cleanup"
Oct 30 01:00:37.525335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.525334    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="020ef144-858f-4125-a9b8-f18fbebbffad" containerName="cleanup"
Oct 30 01:00:37.525521 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.525370    6099 cpu_manager.go:414] "containerMap: removing container" podUID="020ef144-858f-4125-a9b8-f18fbebbffad" containerName="cleanup"
Oct 30 01:00:37.525521 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.525379    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="020ef144-858f-4125-a9b8-f18fbebbffad" containerName="cleanup"
Oct 30 01:00:37.525636 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.525619    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="020ef144-858f-4125-a9b8-f18fbebbffad" containerName="cleanup"
Oct 30 01:00:37.525701 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.525646    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="020ef144-858f-4125-a9b8-f18fbebbffad" containerName="cleanup"
Oct 30 01:00:37.529064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.529043    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd"
Oct 30 01:00:37.547361 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.547333    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-820c04\""
Oct 30 01:00:37.547361 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.547354    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-820c04-dockercfg-7nxjm\""
Oct 30 01:00:37.551904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.551883    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd"]
Oct 30 01:00:37.575876 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.575846    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd"]
Oct 30 01:00:37.580553 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.580533    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd"
Oct 30 01:00:37.664735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.657160    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd"]
Oct 30 01:00:37.664735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.659147    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xx8h\" (UniqueName: \"kubernetes.io/projected/15ed7bd6-34b8-49f2-af6b-453b0c7f921a-kube-api-access-4xx8h\") pod \"payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd\" (UID: \"15ed7bd6-34b8-49f2-af6b-453b0c7f921a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd"
Oct 30 01:00:37.664735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.659436    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15ed7bd6-34b8-49f2-af6b-453b0c7f921a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd\" (UID: \"15ed7bd6-34b8-49f2-af6b-453b0c7f921a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd"
Oct 30 01:00:37.864794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.829261    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15ed7bd6-34b8-49f2-af6b-453b0c7f921a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd\" (UID: \"15ed7bd6-34b8-49f2-af6b-453b0c7f921a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd"
Oct 30 01:00:37.864794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.829374    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk8fh\" (UniqueName: \"kubernetes.io/projected/66824435-d248-47f0-ab98-f8f2c53e25b1-kube-api-access-hk8fh\") pod \"payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd\" (UID: \"66824435-d248-47f0-ab98-f8f2c53e25b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd"
Oct 30 01:00:37.864794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.829500    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xx8h\" (UniqueName: \"kubernetes.io/projected/15ed7bd6-34b8-49f2-af6b-453b0c7f921a-kube-api-access-4xx8h\") pod \"payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd\" (UID: \"15ed7bd6-34b8-49f2-af6b-453b0c7f921a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd"
Oct 30 01:00:37.864794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.829595    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66824435-d248-47f0-ab98-f8f2c53e25b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd\" (UID: \"66824435-d248-47f0-ab98-f8f2c53e25b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd"
Oct 30 01:00:37.864794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.830930    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15ed7bd6-34b8-49f2-af6b-453b0c7f921a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd\" (UID: \"15ed7bd6-34b8-49f2-af6b-453b0c7f921a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd"
Oct 30 01:00:37.889800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.887743    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xx8h\" (UniqueName: \"kubernetes.io/projected/15ed7bd6-34b8-49f2-af6b-453b0c7f921a-kube-api-access-4xx8h\") pod \"payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd\" (UID: \"15ed7bd6-34b8-49f2-af6b-453b0c7f921a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd"
Oct 30 01:00:37.931293 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.931256    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66824435-d248-47f0-ab98-f8f2c53e25b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd\" (UID: \"66824435-d248-47f0-ab98-f8f2c53e25b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd"
Oct 30 01:00:37.931629 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.931606    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk8fh\" (UniqueName: \"kubernetes.io/projected/66824435-d248-47f0-ab98-f8f2c53e25b1-kube-api-access-hk8fh\") pod \"payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd\" (UID: \"66824435-d248-47f0-ab98-f8f2c53e25b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd"
Oct 30 01:00:37.932063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.932039    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66824435-d248-47f0-ab98-f8f2c53e25b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd\" (UID: \"66824435-d248-47f0-ab98-f8f2c53e25b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd"
Oct 30 01:00:37.954519 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:37.954485    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk8fh\" (UniqueName: \"kubernetes.io/projected/66824435-d248-47f0-ab98-f8f2c53e25b1-kube-api-access-hk8fh\") pod \"payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd\" (UID: \"66824435-d248-47f0-ab98-f8f2c53e25b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd"
Oct 30 01:00:38.142266 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:38.142176    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd"
Oct 30 01:00:38.222331 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:38.222238    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd"
Oct 30 01:00:38.273300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:38.273257    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="020ef144-858f-4125-a9b8-f18fbebbffad" path="/var/lib/kubelet/pods/020ef144-858f-4125-a9b8-f18fbebbffad/volumes"
Oct 30 01:00:38.446490 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:38.446462    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ed7bd6_34b8_49f2_af6b_453b0c7f921a.slice/crio-db9adb8f240ca597d2a6e65f9a1a2bf7cfe034da4a94749f496977cfd4da522b WatchSource:0}: Error finding container db9adb8f240ca597d2a6e65f9a1a2bf7cfe034da4a94749f496977cfd4da522b: Status 404 returned error can't find the container with id db9adb8f240ca597d2a6e65f9a1a2bf7cfe034da4a94749f496977cfd4da522b
Oct 30 01:00:38.447828 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:38.447805    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd"]
Oct 30 01:00:38.537717 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:38.537672    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66824435_d248_47f0_ab98_f8f2c53e25b1.slice/crio-b747ad293b17e92fd699e5553057b93e65c0b85e53f8aced9117b8d3c07ce767 WatchSource:0}: Error finding container b747ad293b17e92fd699e5553057b93e65c0b85e53f8aced9117b8d3c07ce767: Status 404 returned error can't find the container with id b747ad293b17e92fd699e5553057b93e65c0b85e53f8aced9117b8d3c07ce767
Oct 30 01:00:38.540542 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:38.540520    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd"]
Oct 30 01:00:38.908027 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:38.907979    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd" event={"ID":"15ed7bd6-34b8-49f2-af6b-453b0c7f921a","Type":"ContainerStarted","Data":"f3cc288764ab2d5194bcba6407fd595cd0826cf2a8fca50a4129504644f9ac26"}
Oct 30 01:00:38.908027 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:38.908038    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd" event={"ID":"15ed7bd6-34b8-49f2-af6b-453b0c7f921a","Type":"ContainerStarted","Data":"db9adb8f240ca597d2a6e65f9a1a2bf7cfe034da4a94749f496977cfd4da522b"}
Oct 30 01:00:38.908288 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:38.908065    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd"
Oct 30 01:00:38.910308 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:38.910282    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd" event={"ID":"66824435-d248-47f0-ab98-f8f2c53e25b1","Type":"ContainerStarted","Data":"c1198035f1cbb5fa1a4e781a9437167344d573b84a79bc5913e84c011c21e342"}
Oct 30 01:00:38.910440 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:38.910360    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd"
Oct 30 01:00:38.910440 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:38.910396    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd" event={"ID":"66824435-d248-47f0-ab98-f8f2c53e25b1","Type":"ContainerStarted","Data":"b747ad293b17e92fd699e5553057b93e65c0b85e53f8aced9117b8d3c07ce767"}
Oct 30 01:00:38.955221 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:38.955174    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd" podStartSLOduration=1.955161197 podStartE2EDuration="1.955161197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:38.95154815 +0000 UTC m=+1532.808455905" watchObservedRunningTime="2026-10-30 01:00:38.955161197 +0000 UTC m=+1532.812068943"
Oct 30 01:00:38.996033 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:38.995981    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd" podStartSLOduration=1.9959674280000002 podStartE2EDuration="1.995967428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:38.995250849 +0000 UTC m=+1532.852158604" watchObservedRunningTime="2026-10-30 01:00:38.995967428 +0000 UTC m=+1532.852875225"
Oct 30 01:00:40.510930 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.510892    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc"]
Oct 30 01:00:40.515346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.515321    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc"
Oct 30 01:00:40.547373 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.547347    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a07920\""
Oct 30 01:00:40.547373 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.547356    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a07920-dockercfg-kflh4\""
Oct 30 01:00:40.551958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.551934    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc"]
Oct 30 01:00:40.575037 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.574997    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc\" (UID: \"ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc"
Oct 30 01:00:40.575353 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.575324    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdjrs\" (UniqueName: \"kubernetes.io/projected/ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f-kube-api-access-kdjrs\") pod \"payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc\" (UID: \"ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc"
Oct 30 01:00:40.657570 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.657521    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw"]
Oct 30 01:00:40.669925 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.667261    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw"
Oct 30 01:00:40.681350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.681313    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdjrs\" (UniqueName: \"kubernetes.io/projected/ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f-kube-api-access-kdjrs\") pod \"payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc\" (UID: \"ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc"
Oct 30 01:00:40.681581 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.681556    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc\" (UID: \"ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc"
Oct 30 01:00:40.682917 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.682891    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc\" (UID: \"ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc"
Oct 30 01:00:40.702936 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.702737    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw"]
Oct 30 01:00:40.730074 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.730021    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdjrs\" (UniqueName: \"kubernetes.io/projected/ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f-kube-api-access-kdjrs\") pod \"payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc\" (UID: \"ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc"
Oct 30 01:00:40.783309 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.783217    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk8qk\" (UniqueName: \"kubernetes.io/projected/17244f1e-2983-4624-83b3-955995424494-kube-api-access-zk8qk\") pod \"payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw\" (UID: \"17244f1e-2983-4624-83b3-955995424494\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw"
Oct 30 01:00:40.783505 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.783311    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17244f1e-2983-4624-83b3-955995424494-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw\" (UID: \"17244f1e-2983-4624-83b3-955995424494\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw"
Oct 30 01:00:40.830073 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.830039    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc"
Oct 30 01:00:40.885265 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.885227    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk8qk\" (UniqueName: \"kubernetes.io/projected/17244f1e-2983-4624-83b3-955995424494-kube-api-access-zk8qk\") pod \"payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw\" (UID: \"17244f1e-2983-4624-83b3-955995424494\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw"
Oct 30 01:00:40.885501 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.885358    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17244f1e-2983-4624-83b3-955995424494-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw\" (UID: \"17244f1e-2983-4624-83b3-955995424494\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw"
Oct 30 01:00:40.886373 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.886342    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17244f1e-2983-4624-83b3-955995424494-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw\" (UID: \"17244f1e-2983-4624-83b3-955995424494\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw"
Oct 30 01:00:40.957231 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:40.955511    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk8qk\" (UniqueName: \"kubernetes.io/projected/17244f1e-2983-4624-83b3-955995424494-kube-api-access-zk8qk\") pod \"payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw\" (UID: \"17244f1e-2983-4624-83b3-955995424494\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw"
Oct 30 01:00:41.010235 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:41.002397    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw"
Oct 30 01:00:41.180502 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:41.180463    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab72b6bc_2c27_4954_b9c3_b26b7fc1f43f.slice/crio-9503fe54e4b4bb90d95f1e351a50c0e70f0beed950a7140d22aa38578db1f73c WatchSource:0}: Error finding container 9503fe54e4b4bb90d95f1e351a50c0e70f0beed950a7140d22aa38578db1f73c: Status 404 returned error can't find the container with id 9503fe54e4b4bb90d95f1e351a50c0e70f0beed950a7140d22aa38578db1f73c
Oct 30 01:00:41.180932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:41.180910    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc"]
Oct 30 01:00:41.381550 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:41.381522    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17244f1e_2983_4624_83b3_955995424494.slice/crio-354ee0e6c67da25eed5757f7d9afc933c939ff9b67a35d363d16317a1672a271 WatchSource:0}: Error finding container 354ee0e6c67da25eed5757f7d9afc933c939ff9b67a35d363d16317a1672a271: Status 404 returned error can't find the container with id 354ee0e6c67da25eed5757f7d9afc933c939ff9b67a35d363d16317a1672a271
Oct 30 01:00:41.382919 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:41.382893    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw"]
Oct 30 01:00:41.965163 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:41.965095    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw" event={"ID":"17244f1e-2983-4624-83b3-955995424494","Type":"ContainerStarted","Data":"7fbc0f38c8852e8cd7551b9baa7ea0b9370def39822d93a4a98b19772ab14bfe"}
Oct 30 01:00:41.965640 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:41.965183    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw"
Oct 30 01:00:41.965640 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:41.965209    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw" event={"ID":"17244f1e-2983-4624-83b3-955995424494","Type":"ContainerStarted","Data":"354ee0e6c67da25eed5757f7d9afc933c939ff9b67a35d363d16317a1672a271"}
Oct 30 01:00:41.967546 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:41.967518    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc" event={"ID":"ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f","Type":"ContainerStarted","Data":"7984101ec73e2d7e256b925fde5ad2f0dfbae19f94cec37e48e2c445e84c9328"}
Oct 30 01:00:41.967671 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:41.967569    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc" event={"ID":"ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f","Type":"ContainerStarted","Data":"9503fe54e4b4bb90d95f1e351a50c0e70f0beed950a7140d22aa38578db1f73c"}
Oct 30 01:00:41.967671 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:41.967598    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc"
Oct 30 01:00:42.009938 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:42.009883    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw" podStartSLOduration=2.009869521 podStartE2EDuration="2.009869521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:42.005623382 +0000 UTC m=+1535.862531137" watchObservedRunningTime="2026-10-30 01:00:42.009869521 +0000 UTC m=+1535.866777276"
Oct 30 01:00:42.046654 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:42.046594    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc" podStartSLOduration=2.046577322 podStartE2EDuration="2.046577322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:42.045633188 +0000 UTC m=+1535.902540942" watchObservedRunningTime="2026-10-30 01:00:42.046577322 +0000 UTC m=+1535.903485077"
Oct 30 01:00:45.375969 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.375935    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"]
Oct 30 01:00:45.380856 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.380836    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.408105 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.408080    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"]
Oct 30 01:00:45.474284 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.474243    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-token\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.474465 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.474409    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-envoy\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.474465 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.474444    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-data\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.475812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.474581    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-workload-socket\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.475812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.474680    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-credential-socket\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.475812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.474925    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x7x6\" (UniqueName: \"kubernetes.io/projected/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-kube-api-access-2x7x6\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.475812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.475046    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-podinfo\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.475812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.475108    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-workload-certs\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.475812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.475162    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istiod-ca-cert\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.480341 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.480295    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2a8b26-openshift-default-dockercfg-gn4zc\""
Oct 30 01:00:45.591727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.591439    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-credential-socket\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.597729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.594071    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-credential-socket\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.597729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.594231    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x7x6\" (UniqueName: \"kubernetes.io/projected/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-kube-api-access-2x7x6\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.597729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.594370    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-podinfo\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.597729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.594443    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-workload-certs\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.597729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.594501    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istiod-ca-cert\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.597729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.594650    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-token\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.597729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.594811    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-envoy\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.597729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.594838    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-data\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.597729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.594884    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-workload-socket\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.597729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.595374    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-workload-socket\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.597729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.596167    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istiod-ca-cert\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.597729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.597181    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-data\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.597729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.597699    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-workload-certs\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.601046 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.600993    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-podinfo\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.616968 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.616926    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-envoy\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.636511 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.636430    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x7x6\" (UniqueName: \"kubernetes.io/projected/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-kube-api-access-2x7x6\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.641764 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.639418    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-token\") pod \"e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") " pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:45.696032 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:45.696000    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:46.040495 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:46.040459    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a34143_9d52_4d6a_99d3_05d6501f1bb7.slice/crio-d326bfff99249bcd76becca7d06ef0fa6abe01543c850be4bc758995a8fa298e WatchSource:0}: Error finding container d326bfff99249bcd76becca7d06ef0fa6abe01543c850be4bc758995a8fa298e: Status 404 returned error can't find the container with id d326bfff99249bcd76becca7d06ef0fa6abe01543c850be4bc758995a8fa298e
Oct 30 01:00:46.040638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:46.040597    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"]
Oct 30 01:00:46.043474 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:46.043437    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:46.043593 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:46.043511    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:46.043593 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:46.043547    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:46.632546 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:00:46.632512    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:00:47.054631 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:47.054590    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf" event={"ID":"c9a34143-9d52-4d6a-99d3-05d6501f1bb7","Type":"ContainerStarted","Data":"86971e1ba50bbb98ffa4e637644d94ed90d0edbc81b42f053f4584241f82a27a"}
Oct 30 01:00:47.054864 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:47.054670    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf" event={"ID":"c9a34143-9d52-4d6a-99d3-05d6501f1bb7","Type":"ContainerStarted","Data":"d326bfff99249bcd76becca7d06ef0fa6abe01543c850be4bc758995a8fa298e"}
Oct 30 01:00:47.699069 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:47.699029    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:47.718837 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:47.718802    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:47.769121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:47.769058    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf" podStartSLOduration=2.769039892 podStartE2EDuration="2.769039892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:47.129674503 +0000 UTC m=+1540.986582259" watchObservedRunningTime="2026-10-30 01:00:47.769039892 +0000 UTC m=+1541.625947647"
Oct 30 01:00:48.063314 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:48.063278    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:48.064373 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:48.064351    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:00:48.565673 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:48.565639    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26"]
Oct 30 01:00:48.570474 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:48.570457    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26"
Oct 30 01:00:48.582447 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:48.582422    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b77c32-727e-41e1-9303-5262f1b1e3b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-866af8b6-2xs26\" (UID: \"b6b77c32-727e-41e1-9303-5262f1b1e3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26"
Oct 30 01:00:48.582552 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:48.582540    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kznw8\" (UniqueName: \"kubernetes.io/projected/b6b77c32-727e-41e1-9303-5262f1b1e3b0-kube-api-access-kznw8\") pod \"maas-api-revoke-sub-866af8b6-2xs26\" (UID: \"b6b77c32-727e-41e1-9303-5262f1b1e3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26"
Oct 30 01:00:48.582642 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:48.582625    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b77c32-727e-41e1-9303-5262f1b1e3b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-866af8b6-2xs26\" (UID: \"b6b77c32-727e-41e1-9303-5262f1b1e3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26"
Oct 30 01:00:48.589806 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:48.589782    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26"]
Oct 30 01:00:48.590010 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:48.589993    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:00:48.590127 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:48.590109    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:00:48.684530 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:48.684498    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kznw8\" (UniqueName: \"kubernetes.io/projected/b6b77c32-727e-41e1-9303-5262f1b1e3b0-kube-api-access-kznw8\") pod \"maas-api-revoke-sub-866af8b6-2xs26\" (UID: \"b6b77c32-727e-41e1-9303-5262f1b1e3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26"
Oct 30 01:00:48.684738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:48.684571    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b77c32-727e-41e1-9303-5262f1b1e3b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-866af8b6-2xs26\" (UID: \"b6b77c32-727e-41e1-9303-5262f1b1e3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26"
Oct 30 01:00:48.684785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:48.684768    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b77c32-727e-41e1-9303-5262f1b1e3b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-866af8b6-2xs26\" (UID: \"b6b77c32-727e-41e1-9303-5262f1b1e3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26"
Oct 30 01:00:48.685455 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:48.685431    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b77c32-727e-41e1-9303-5262f1b1e3b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-866af8b6-2xs26\" (UID: \"b6b77c32-727e-41e1-9303-5262f1b1e3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26"
Oct 30 01:00:48.687108 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:48.687080    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b77c32-727e-41e1-9303-5262f1b1e3b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-866af8b6-2xs26\" (UID: \"b6b77c32-727e-41e1-9303-5262f1b1e3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26"
Oct 30 01:00:48.708201 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:48.708178    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kznw8\" (UniqueName: \"kubernetes.io/projected/b6b77c32-727e-41e1-9303-5262f1b1e3b0-kube-api-access-kznw8\") pod \"maas-api-revoke-sub-866af8b6-2xs26\" (UID: \"b6b77c32-727e-41e1-9303-5262f1b1e3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26"
Oct 30 01:00:48.885972 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:48.885894    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26"
Oct 30 01:00:49.217580 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:49.217544    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b77c32_727e_41e1_9303_5262f1b1e3b0.slice/crio-19da08fb81b6b2440762944cdaee70f4600403a51c3e4ba6713e8072551c4447 WatchSource:0}: Error finding container 19da08fb81b6b2440762944cdaee70f4600403a51c3e4ba6713e8072551c4447: Status 404 returned error can't find the container with id 19da08fb81b6b2440762944cdaee70f4600403a51c3e4ba6713e8072551c4447
Oct 30 01:00:49.222744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:49.222718    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26"]
Oct 30 01:00:49.922395 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:49.922325    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd"
Oct 30 01:00:49.922395 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:49.922392    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd"
Oct 30 01:00:50.100499 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:50.100456    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26" event={"ID":"b6b77c32-727e-41e1-9303-5262f1b1e3b0","Type":"ContainerStarted","Data":"f02cc3d854d25a2bf0238e62ab2ec4717de3ee72cc510fc8d7349735d8273c09"}
Oct 30 01:00:50.100734 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:50.100574    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26" event={"ID":"b6b77c32-727e-41e1-9303-5262f1b1e3b0","Type":"ContainerStarted","Data":"19da08fb81b6b2440762944cdaee70f4600403a51c3e4ba6713e8072551c4447"}
Oct 30 01:00:51.103937 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:51.103904    6099 generic.go:356] "Generic (PLEG): container finished" podID="b6b77c32-727e-41e1-9303-5262f1b1e3b0" containerID="f02cc3d854d25a2bf0238e62ab2ec4717de3ee72cc510fc8d7349735d8273c09" exitCode=0
Oct 30 01:00:51.104302 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:51.103979    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26" event={"ID":"b6b77c32-727e-41e1-9303-5262f1b1e3b0","Type":"ContainerDied","Data":"f02cc3d854d25a2bf0238e62ab2ec4717de3ee72cc510fc8d7349735d8273c09"}
Oct 30 01:00:51.989690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:51.989641    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz"]
Oct 30 01:00:51.994360 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:51.994336    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz"
Oct 30 01:00:52.013883 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.013862    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2a8b26-dockercfg-sjncz\""
Oct 30 01:00:52.015331 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.015306    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n"]
Oct 30 01:00:52.020964 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.020943    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n"
Oct 30 01:00:52.045927 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.045892    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz"]
Oct 30 01:00:52.049734 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.049673    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwf2b\" (UniqueName: \"kubernetes.io/projected/3f9f6975-a3df-4ed1-940a-a801424eec23-kube-api-access-zwf2b\") pod \"payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz\" (UID: \"3f9f6975-a3df-4ed1-940a-a801424eec23\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz"
Oct 30 01:00:52.050164 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.050141    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f9f6975-a3df-4ed1-940a-a801424eec23-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz\" (UID: \"3f9f6975-a3df-4ed1-940a-a801424eec23\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz"
Oct 30 01:00:52.050417 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.050398    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e85af1-92a4-4f4e-8ef3-5cce4c583000-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n\" (UID: \"84e85af1-92a4-4f4e-8ef3-5cce4c583000\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n"
Oct 30 01:00:52.050617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.050597    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d26dc\" (UniqueName: \"kubernetes.io/projected/84e85af1-92a4-4f4e-8ef3-5cce4c583000-kube-api-access-d26dc\") pod \"payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n\" (UID: \"84e85af1-92a4-4f4e-8ef3-5cce4c583000\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n"
Oct 30 01:00:52.084251 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.079217    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2a8b26\""
Oct 30 01:00:52.154824 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.141505    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n"]
Oct 30 01:00:52.155262 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.154910    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwf2b\" (UniqueName: \"kubernetes.io/projected/3f9f6975-a3df-4ed1-940a-a801424eec23-kube-api-access-zwf2b\") pod \"payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz\" (UID: \"3f9f6975-a3df-4ed1-940a-a801424eec23\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz"
Oct 30 01:00:52.155262 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.155042    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f9f6975-a3df-4ed1-940a-a801424eec23-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz\" (UID: \"3f9f6975-a3df-4ed1-940a-a801424eec23\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz"
Oct 30 01:00:52.155262 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.155125    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e85af1-92a4-4f4e-8ef3-5cce4c583000-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n\" (UID: \"84e85af1-92a4-4f4e-8ef3-5cce4c583000\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n"
Oct 30 01:00:52.155262 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.155200    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d26dc\" (UniqueName: \"kubernetes.io/projected/84e85af1-92a4-4f4e-8ef3-5cce4c583000-kube-api-access-d26dc\") pod \"payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n\" (UID: \"84e85af1-92a4-4f4e-8ef3-5cce4c583000\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n"
Oct 30 01:00:52.170740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.156735    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f9f6975-a3df-4ed1-940a-a801424eec23-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz\" (UID: \"3f9f6975-a3df-4ed1-940a-a801424eec23\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz"
Oct 30 01:00:52.170740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.157559    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e85af1-92a4-4f4e-8ef3-5cce4c583000-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n\" (UID: \"84e85af1-92a4-4f4e-8ef3-5cce4c583000\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n"
Oct 30 01:00:52.195400 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.195365    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwf2b\" (UniqueName: \"kubernetes.io/projected/3f9f6975-a3df-4ed1-940a-a801424eec23-kube-api-access-zwf2b\") pod \"payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz\" (UID: \"3f9f6975-a3df-4ed1-940a-a801424eec23\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz"
Oct 30 01:00:52.199855 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.199830    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d26dc\" (UniqueName: \"kubernetes.io/projected/84e85af1-92a4-4f4e-8ef3-5cce4c583000-kube-api-access-d26dc\") pod \"payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n\" (UID: \"84e85af1-92a4-4f4e-8ef3-5cce4c583000\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n"
Oct 30 01:00:52.308912 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.308608    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz"
Oct 30 01:00:52.486327 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.486293    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n"
Oct 30 01:00:52.532690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.532653    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26"
Oct 30 01:00:52.569173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.569138    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b77c32-727e-41e1-9303-5262f1b1e3b0-kube-api-access-kznw8\" (UniqueName: \"kubernetes.io/projected/b6b77c32-727e-41e1-9303-5262f1b1e3b0-kube-api-access-kznw8\") pod \"b6b77c32-727e-41e1-9303-5262f1b1e3b0\" (UID: \"b6b77c32-727e-41e1-9303-5262f1b1e3b0\") "
Oct 30 01:00:52.569889 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.569859    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6b77c32-727e-41e1-9303-5262f1b1e3b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b77c32-727e-41e1-9303-5262f1b1e3b0-maas-api-xks-ca\") pod \"b6b77c32-727e-41e1-9303-5262f1b1e3b0\" (UID: \"b6b77c32-727e-41e1-9303-5262f1b1e3b0\") "
Oct 30 01:00:52.570062 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.570030    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b77c32-727e-41e1-9303-5262f1b1e3b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b77c32-727e-41e1-9303-5262f1b1e3b0-maas-api-openshift-ca\") pod \"b6b77c32-727e-41e1-9303-5262f1b1e3b0\" (UID: \"b6b77c32-727e-41e1-9303-5262f1b1e3b0\") "
Oct 30 01:00:52.570476 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.570451    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b77c32-727e-41e1-9303-5262f1b1e3b0-maas-api-openshift-ca" pod "b6b77c32-727e-41e1-9303-5262f1b1e3b0" (UID: "b6b77c32-727e-41e1-9303-5262f1b1e3b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:52.570990 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.570959    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b77c32-727e-41e1-9303-5262f1b1e3b0-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:52.571834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.571628    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b77c32-727e-41e1-9303-5262f1b1e3b0-kube-api-access-kznw8" pod "b6b77c32-727e-41e1-9303-5262f1b1e3b0" (UID: "b6b77c32-727e-41e1-9303-5262f1b1e3b0"). InnerVolumeSpecName "kube-api-access-kznw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:52.572007 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.571986    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b77c32-727e-41e1-9303-5262f1b1e3b0-maas-api-xks-ca" pod "b6b77c32-727e-41e1-9303-5262f1b1e3b0" (UID: "b6b77c32-727e-41e1-9303-5262f1b1e3b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:52.673054 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.673017    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kznw8\" (UniqueName: \"kubernetes.io/projected/b6b77c32-727e-41e1-9303-5262f1b1e3b0-kube-api-access-kznw8\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:52.673054 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.673055    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b77c32-727e-41e1-9303-5262f1b1e3b0-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:00:52.692935 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:52.692902    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9f6975_a3df_4ed1_940a_a801424eec23.slice/crio-eb94b09e9afea31cd1438b644f6c0b37490270d20e230ed170ce52d5ac859e41 WatchSource:0}: Error finding container eb94b09e9afea31cd1438b644f6c0b37490270d20e230ed170ce52d5ac859e41: Status 404 returned error can't find the container with id eb94b09e9afea31cd1438b644f6c0b37490270d20e230ed170ce52d5ac859e41
Oct 30 01:00:52.693676 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.693650    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz"]
Oct 30 01:00:52.811680 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:00:52.811651    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e85af1_92a4_4f4e_8ef3_5cce4c583000.slice/crio-50ba7c465f33b6e76f2be7e6d93798165caa0d461b382e8bd085a2e451d30752 WatchSource:0}: Error finding container 50ba7c465f33b6e76f2be7e6d93798165caa0d461b382e8bd085a2e451d30752: Status 404 returned error can't find the container with id 50ba7c465f33b6e76f2be7e6d93798165caa0d461b382e8bd085a2e451d30752
Oct 30 01:00:52.814070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:52.814045    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n"]
Oct 30 01:00:53.011766 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:53.011728    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc"
Oct 30 01:00:53.011950 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:53.011825    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw"
Oct 30 01:00:53.300464 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:53.300429    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26"
Oct 30 01:00:53.300972 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:53.300419    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26" event={"ID":"b6b77c32-727e-41e1-9303-5262f1b1e3b0","Type":"ContainerDied","Data":"19da08fb81b6b2440762944cdaee70f4600403a51c3e4ba6713e8072551c4447"}
Oct 30 01:00:53.300972 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:53.300558    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19da08fb81b6b2440762944cdaee70f4600403a51c3e4ba6713e8072551c4447"
Oct 30 01:00:53.302843 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:53.302815    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz" event={"ID":"3f9f6975-a3df-4ed1-940a-a801424eec23","Type":"ContainerStarted","Data":"ffcb1f485dacf1b42b6f911f768adb7f4ebb75d544d3a94e4c4bc834f5fefe30"}
Oct 30 01:00:53.302986 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:53.302874    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz" event={"ID":"3f9f6975-a3df-4ed1-940a-a801424eec23","Type":"ContainerStarted","Data":"eb94b09e9afea31cd1438b644f6c0b37490270d20e230ed170ce52d5ac859e41"}
Oct 30 01:00:53.302986 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:53.302909    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz"
Oct 30 01:00:53.304917 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:53.304862    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n" event={"ID":"84e85af1-92a4-4f4e-8ef3-5cce4c583000","Type":"ContainerStarted","Data":"7d5fa23e632ed13747458c51b661c9cd8cda95e49cc1bed5cae798fa802713de"}
Oct 30 01:00:53.304917 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:53.304913    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n" event={"ID":"84e85af1-92a4-4f4e-8ef3-5cce4c583000","Type":"ContainerStarted","Data":"50ba7c465f33b6e76f2be7e6d93798165caa0d461b382e8bd085a2e451d30752"}
Oct 30 01:00:53.305260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:53.304983    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n"
Oct 30 01:00:53.358800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:53.358738    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz" podStartSLOduration=2.358701086 podStartE2EDuration="2.358701086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:53.354553021 +0000 UTC m=+1547.211460778" watchObservedRunningTime="2026-10-30 01:00:53.358701086 +0000 UTC m=+1547.215608843"
Oct 30 01:00:53.400261 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:00:53.400204    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n" podStartSLOduration=2.400189838 podStartE2EDuration="2.400189838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:53.398315347 +0000 UTC m=+1547.255223099" watchObservedRunningTime="2026-10-30 01:00:53.400189838 +0000 UTC m=+1547.257097592"
Oct 30 01:01:04.316607 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:04.316571    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n"
Oct 30 01:01:04.317103 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:04.316645    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz"
Oct 30 01:01:06.188917 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.188881    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9"]
Oct 30 01:01:06.191444 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.191409    6099 cpu_manager.go:414] "containerMap: removing container" podUID="b6b77c32-727e-41e1-9303-5262f1b1e3b0" containerName="revoke-keys"
Oct 30 01:01:06.191444 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.191443    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b77c32-727e-41e1-9303-5262f1b1e3b0" containerName="revoke-keys"
Oct 30 01:01:06.191886 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.191865    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b77c32-727e-41e1-9303-5262f1b1e3b0" containerName="revoke-keys"
Oct 30 01:01:06.197956 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.197932    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9"
Oct 30 01:01:06.219440 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.219409    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-969b92\""
Oct 30 01:01:06.252786 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.250994    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-969b92-dockercfg-4bfs6\""
Oct 30 01:01:06.351870 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.330887    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b90542e-141d-47e4-8270-4aa8e67632ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9\" (UID: \"6b90542e-141d-47e4-8270-4aa8e67632ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9"
Oct 30 01:01:06.351870 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.331229    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jppb\" (UniqueName: \"kubernetes.io/projected/6b90542e-141d-47e4-8270-4aa8e67632ca-kube-api-access-4jppb\") pod \"payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9\" (UID: \"6b90542e-141d-47e4-8270-4aa8e67632ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9"
Oct 30 01:01:06.433307 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.432782    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jppb\" (UniqueName: \"kubernetes.io/projected/6b90542e-141d-47e4-8270-4aa8e67632ca-kube-api-access-4jppb\") pod \"payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9\" (UID: \"6b90542e-141d-47e4-8270-4aa8e67632ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9"
Oct 30 01:01:06.433940 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.433913    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b90542e-141d-47e4-8270-4aa8e67632ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9\" (UID: \"6b90542e-141d-47e4-8270-4aa8e67632ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9"
Oct 30 01:01:06.453030 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.452946    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-969b92\""
Oct 30 01:01:06.455249 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.455205    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b90542e-141d-47e4-8270-4aa8e67632ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9\" (UID: \"6b90542e-141d-47e4-8270-4aa8e67632ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9"
Oct 30 01:01:06.456408 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.456339    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9"]
Oct 30 01:01:06.460211 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.460168    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq"]
Oct 30 01:01:06.466377 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.466333    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq"
Oct 30 01:01:06.473135 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.473100    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq"]
Oct 30 01:01:06.473660 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.473633    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jppb\" (UniqueName: \"kubernetes.io/projected/6b90542e-141d-47e4-8270-4aa8e67632ca-kube-api-access-4jppb\") pod \"payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9\" (UID: \"6b90542e-141d-47e4-8270-4aa8e67632ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9"
Oct 30 01:01:06.525086 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.525050    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9"
Oct 30 01:01:06.536542 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.536506    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx8h8\" (UniqueName: \"kubernetes.io/projected/6adc9cb6-daf1-4c37-b49f-96cfaf6a8387-kube-api-access-lx8h8\") pod \"payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq\" (UID: \"6adc9cb6-daf1-4c37-b49f-96cfaf6a8387\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq"
Oct 30 01:01:06.536988 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.536965    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6adc9cb6-daf1-4c37-b49f-96cfaf6a8387-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq\" (UID: \"6adc9cb6-daf1-4c37-b49f-96cfaf6a8387\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq"
Oct 30 01:01:06.644830 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.638300    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6adc9cb6-daf1-4c37-b49f-96cfaf6a8387-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq\" (UID: \"6adc9cb6-daf1-4c37-b49f-96cfaf6a8387\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq"
Oct 30 01:01:06.644830 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.638473    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx8h8\" (UniqueName: \"kubernetes.io/projected/6adc9cb6-daf1-4c37-b49f-96cfaf6a8387-kube-api-access-lx8h8\") pod \"payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq\" (UID: \"6adc9cb6-daf1-4c37-b49f-96cfaf6a8387\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq"
Oct 30 01:01:06.644830 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.639848    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6adc9cb6-daf1-4c37-b49f-96cfaf6a8387-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq\" (UID: \"6adc9cb6-daf1-4c37-b49f-96cfaf6a8387\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq"
Oct 30 01:01:06.671534 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.671503    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx8h8\" (UniqueName: \"kubernetes.io/projected/6adc9cb6-daf1-4c37-b49f-96cfaf6a8387-kube-api-access-lx8h8\") pod \"payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq\" (UID: \"6adc9cb6-daf1-4c37-b49f-96cfaf6a8387\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq"
Oct 30 01:01:06.802586 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.802555    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq"
Oct 30 01:01:06.842978 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:01:06.842944    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b90542e_141d_47e4_8270_4aa8e67632ca.slice/crio-a41f1cf6ccb6afa41bb9bf58ea75ab2b413f058608ca512c7848596e80dfe1ba WatchSource:0}: Error finding container a41f1cf6ccb6afa41bb9bf58ea75ab2b413f058608ca512c7848596e80dfe1ba: Status 404 returned error can't find the container with id a41f1cf6ccb6afa41bb9bf58ea75ab2b413f058608ca512c7848596e80dfe1ba
Oct 30 01:01:06.844944 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:06.844914    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9"]
Oct 30 01:01:07.166646 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:07.166617    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq"]
Oct 30 01:01:07.166802 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:01:07.166648    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6adc9cb6_daf1_4c37_b49f_96cfaf6a8387.slice/crio-9612fd02cbe7c0ce581456b03aaec4d515dbfbc8bc3e7e728e4ddc3b006e7eb5 WatchSource:0}: Error finding container 9612fd02cbe7c0ce581456b03aaec4d515dbfbc8bc3e7e728e4ddc3b006e7eb5: Status 404 returned error can't find the container with id 9612fd02cbe7c0ce581456b03aaec4d515dbfbc8bc3e7e728e4ddc3b006e7eb5
Oct 30 01:01:07.480475 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:07.480369    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9" event={"ID":"6b90542e-141d-47e4-8270-4aa8e67632ca","Type":"ContainerStarted","Data":"a2fc4d297acb093a1d6c25691d94c54589074f387be1260ecbf063c85ec8f88d"}
Oct 30 01:01:07.480475 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:07.480440    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9" event={"ID":"6b90542e-141d-47e4-8270-4aa8e67632ca","Type":"ContainerStarted","Data":"a41f1cf6ccb6afa41bb9bf58ea75ab2b413f058608ca512c7848596e80dfe1ba"}
Oct 30 01:01:07.480475 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:07.480472    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9"
Oct 30 01:01:07.482304 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:07.482277    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq" event={"ID":"6adc9cb6-daf1-4c37-b49f-96cfaf6a8387","Type":"ContainerStarted","Data":"29e1ae1ddaecfe7808b1edbab9ed6f73d8303ecb59bec59eff0839d46ada3a6d"}
Oct 30 01:01:07.482446 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:07.482322    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq" event={"ID":"6adc9cb6-daf1-4c37-b49f-96cfaf6a8387","Type":"ContainerStarted","Data":"9612fd02cbe7c0ce581456b03aaec4d515dbfbc8bc3e7e728e4ddc3b006e7eb5"}
Oct 30 01:01:07.482446 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:07.482363    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq"
Oct 30 01:01:07.529123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:07.529069    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9" podStartSLOduration=1.529056763 podStartE2EDuration="1.529056763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:07.523425871 +0000 UTC m=+1561.380333629" watchObservedRunningTime="2026-10-30 01:01:07.529056763 +0000 UTC m=+1561.385964517"
Oct 30 01:01:07.568659 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:07.568608    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq" podStartSLOduration=1.568594214 podStartE2EDuration="1.568594214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:07.5660291 +0000 UTC m=+1561.422936856" watchObservedRunningTime="2026-10-30 01:01:07.568594214 +0000 UTC m=+1561.425501970"
Oct 30 01:01:08.696286 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:08.696251    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq"]
Oct 30 01:01:08.711004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:08.710971    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601aea-nzcvq"]
Oct 30 01:01:08.868728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:08.865377    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:01:10.277729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:10.269756    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c60cba8a-34d4-454c-8405-9afa8c9b4f76" path="/var/lib/kubelet/pods/c60cba8a-34d4-454c-8405-9afa8c9b4f76/volumes"
Oct 30 01:01:10.796740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:10.790018    6099 scope.go:122] "RemoveContainer" containerID="f80244b1720ef86826d2356ae1d02200729cdcf82055d9abbd4b93957d4707ce"
Oct 30 01:01:10.958978 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:10.958938    6099 scope.go:122] "RemoveContainer" containerID="b6f56225a0ad2cf1b439e3315559a3a236027e8cf7d1212232795fe5ceb326bd"
Oct 30 01:01:12.975728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:12.971078    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:13.001415 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:13.000997    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:13.302851 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:13.302800    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:13.429735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:13.429635    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:13.455779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:13.445694    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:13.634902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:13.634791    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:13.698731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:13.670218    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:13.752743 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:13.752683    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 01:01:13.754852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:13.754819    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:13.756189 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:13.755344    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 01:01:13.896138 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:13.896034    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:14.023610 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.023567    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:14.025226 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.025201    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 01:01:14.403837 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.403788    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 01:01:14.472744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.464820    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:14.494768 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.494727    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 01:01:14.588732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.568877    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 01:01:14.588732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.568948    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 01:01:14.588732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.568987    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 01:01:14.672786 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.637680    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:14.672786 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.638225    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 01:01:14.753895 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.753837    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:14.926791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.754785    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 01:01:14.926791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.755864    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:01:14.926791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.756318    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 01:01:14.926791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.777286    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": context deadline exceeded"
Oct 30 01:01:14.926791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.777699    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 01:01:14.926791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.791782    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 01:01:14.926791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.792479    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 01:01:14.971731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.939624    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 01:01:14.997745 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:14.972195    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 01:01:15.025404 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:15.025357    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:15.060813 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:15.060770    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn"]
Oct 30 01:01:15.185500 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:15.183372    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 01:01:15.185500 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:15.184313    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:15.239608 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:15.231127    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a32412-w6dtn"]
Oct 30 01:01:15.349727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:15.346232    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 01:01:15.410731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:15.408778    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:15.415735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:15.415306    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 01:01:15.430755 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:15.428534    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 01:01:15.452016 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:15.451936    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 01:01:15.452016 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:15.451998    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 01:01:15.499730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:15.495185    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 01:01:15.499730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:15.496346    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:15.513876 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:15.513783    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 01:01:15.521783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:15.521683    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 01:01:15.523300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:15.522816    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 01:01:16.287731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:16.265342    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6370d09b-ecc5-4ca5-968f-d39f3bdefcde" path="/var/lib/kubelet/pods/6370d09b-ecc5-4ca5-968f-d39f3bdefcde/volumes"
Oct 30 01:01:16.641672 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:01:16.641636    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:01:17.843632 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:17.843591    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"]
Oct 30 01:01:17.844154 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:17.843865    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799" podUID="90b346c4-cc85-4fd7-9578-0533e8e902a9" containerName="istio-proxy" containerID="cri-o://97f29da7df242a4b2f87af0838c1102f353873dab06d5d06c31431d2559210cc" gracePeriod=30
Oct 30 01:01:18.494471 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:18.494443    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq"
Oct 30 01:01:18.494643 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:18.494510    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9"
Oct 30 01:01:20.850867 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:20.850830    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"]
Oct 30 01:01:20.851450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:20.851061    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp" podUID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" containerName="istio-proxy" containerID="cri-o://9a861b5c79b98757e8b1994dd7cde16f934741c92027facdec5bd733794b6b98" gracePeriod=30
Oct 30 01:01:21.133273 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.133195    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"]
Oct 30 01:01:21.133430 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.133394    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl" podUID="90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" containerName="istio-proxy" containerID="cri-o://00485a7d725182467e342da4f7072ea674d22ace36ac7e5c20acfe157bc6b74f" gracePeriod=30
Oct 30 01:01:21.195882 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.195845    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"]
Oct 30 01:01:21.216939 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.216907    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"
Oct 30 01:01:21.240250 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.240216    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a07920-serving-cert\""
Oct 30 01:01:21.241643 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.241618    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"]
Oct 30 01:01:21.332081 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.332044    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5xg8\" (UniqueName: \"kubernetes.io/projected/c789663e-00c2-4c17-9b12-784459a7a9e4-kube-api-access-c5xg8\") pod \"maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh\" (UID: \"c789663e-00c2-4c17-9b12-784459a7a9e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"
Oct 30 01:01:21.332251 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.332166    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c789663e-00c2-4c17-9b12-784459a7a9e4-maas-api-tls\") pod \"maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh\" (UID: \"c789663e-00c2-4c17-9b12-784459a7a9e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"
Oct 30 01:01:21.332362 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.332338    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c789663e-00c2-4c17-9b12-784459a7a9e4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh\" (UID: \"c789663e-00c2-4c17-9b12-784459a7a9e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"
Oct 30 01:01:21.332470 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.332456    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c789663e-00c2-4c17-9b12-784459a7a9e4-metrics-certs\") pod \"maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh\" (UID: \"c789663e-00c2-4c17-9b12-784459a7a9e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"
Oct 30 01:01:21.434652 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.434564    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5xg8\" (UniqueName: \"kubernetes.io/projected/c789663e-00c2-4c17-9b12-784459a7a9e4-kube-api-access-c5xg8\") pod \"maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh\" (UID: \"c789663e-00c2-4c17-9b12-784459a7a9e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"
Oct 30 01:01:21.435012 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.434988    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c789663e-00c2-4c17-9b12-784459a7a9e4-maas-api-tls\") pod \"maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh\" (UID: \"c789663e-00c2-4c17-9b12-784459a7a9e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"
Oct 30 01:01:21.435253 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.435233    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c789663e-00c2-4c17-9b12-784459a7a9e4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh\" (UID: \"c789663e-00c2-4c17-9b12-784459a7a9e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"
Oct 30 01:01:21.435432 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.435415    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c789663e-00c2-4c17-9b12-784459a7a9e4-metrics-certs\") pod \"maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh\" (UID: \"c789663e-00c2-4c17-9b12-784459a7a9e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"
Oct 30 01:01:21.436587 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.436561    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c789663e-00c2-4c17-9b12-784459a7a9e4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh\" (UID: \"c789663e-00c2-4c17-9b12-784459a7a9e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"
Oct 30 01:01:21.462775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.452353    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c789663e-00c2-4c17-9b12-784459a7a9e4-metrics-certs\") pod \"maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh\" (UID: \"c789663e-00c2-4c17-9b12-784459a7a9e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"
Oct 30 01:01:21.463055 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.458858    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c789663e-00c2-4c17-9b12-784459a7a9e4-maas-api-tls\") pod \"maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh\" (UID: \"c789663e-00c2-4c17-9b12-784459a7a9e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"
Oct 30 01:01:21.473830 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.468436    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5xg8\" (UniqueName: \"kubernetes.io/projected/c789663e-00c2-4c17-9b12-784459a7a9e4-kube-api-access-c5xg8\") pod \"maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh\" (UID: \"c789663e-00c2-4c17-9b12-784459a7a9e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"
Oct 30 01:01:21.556090 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.556055    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"
Oct 30 01:01:21.907574 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:01:21.907543    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc789663e_00c2_4c17_9b12_784459a7a9e4.slice/crio-b9961b2ff1939bfb2775bcd68987e24364ec0c0ce4f54f8cb9f32c04a891d984 WatchSource:0}: Error finding container b9961b2ff1939bfb2775bcd68987e24364ec0c0ce4f54f8cb9f32c04a891d984: Status 404 returned error can't find the container with id b9961b2ff1939bfb2775bcd68987e24364ec0c0ce4f54f8cb9f32c04a891d984
Oct 30 01:01:21.909059 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:21.909038    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"]
Oct 30 01:01:22.201764 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:22.201661    6099 generic.go:356] "Generic (PLEG): container finished" podID="58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" containerID="8999a5abb33e86d850a8a12f4f903b3d49b2ee8ed22cdb3834419a048758b9fd" exitCode=0
Oct 30 01:01:22.201764 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:22.201738    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm" event={"ID":"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd","Type":"ContainerDied","Data":"8999a5abb33e86d850a8a12f4f903b3d49b2ee8ed22cdb3834419a048758b9fd"}
Oct 30 01:01:22.202210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:22.202186    6099 scope.go:122] "RemoveContainer" containerID="8999a5abb33e86d850a8a12f4f903b3d49b2ee8ed22cdb3834419a048758b9fd"
Oct 30 01:01:22.203400 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:22.203377    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh" event={"ID":"c789663e-00c2-4c17-9b12-784459a7a9e4","Type":"ContainerStarted","Data":"b9961b2ff1939bfb2775bcd68987e24364ec0c0ce4f54f8cb9f32c04a891d984"}
Oct 30 01:01:22.204084 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:22.204056    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:01:22.204485 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:22.204127    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:01:22.204485 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:22.204162    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:01:23.194975 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.194931    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 01:01:23.214438 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.214397    6099 generic.go:356] "Generic (PLEG): container finished" podID="90b346c4-cc85-4fd7-9578-0533e8e902a9" containerID="97f29da7df242a4b2f87af0838c1102f353873dab06d5d06c31431d2559210cc" exitCode=0
Oct 30 01:01:23.214630 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.214471    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799" event={"ID":"90b346c4-cc85-4fd7-9578-0533e8e902a9","Type":"ContainerDied","Data":"97f29da7df242a4b2f87af0838c1102f353873dab06d5d06c31431d2559210cc"}
Oct 30 01:01:23.214630 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.214486    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"
Oct 30 01:01:23.214630 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.214550    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799" event={"ID":"90b346c4-cc85-4fd7-9578-0533e8e902a9","Type":"ContainerDied","Data":"700e552c258fecf92aab6a9c78fa9f16a79571b3593e63853bf128da32adf647"}
Oct 30 01:01:23.214630 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.214574    6099 scope.go:122] "RemoveContainer" containerID="97f29da7df242a4b2f87af0838c1102f353873dab06d5d06c31431d2559210cc"
Oct 30 01:01:23.217092 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.217070    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm" event={"ID":"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd","Type":"ContainerStarted","Data":"90347dd1d6ea0ddd5d0d14c4faf3c82eceff97b3c18fe4dc47c2435813245d50"}
Oct 30 01:01:23.218926 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.218908    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh" event={"ID":"c789663e-00c2-4c17-9b12-784459a7a9e4","Type":"ContainerStarted","Data":"6f35717d8a81bacb31738c795694a9b52915457328b1ab3248ef4ec8cdd4b343"}
Oct 30 01:01:23.219040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.218985    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"
Oct 30 01:01:23.226275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.226256    6099 scope.go:122] "RemoveContainer" containerID="1652164599eab84070cc7219bc6df07fdc449c1c9cfa7661091ff2acc9113ac7"
Oct 30 01:01:23.236624 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.236609    6099 scope.go:122] "RemoveContainer" containerID="97f29da7df242a4b2f87af0838c1102f353873dab06d5d06c31431d2559210cc"
Oct 30 01:01:23.236915 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:01:23.236892    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97f29da7df242a4b2f87af0838c1102f353873dab06d5d06c31431d2559210cc\": container with ID starting with 97f29da7df242a4b2f87af0838c1102f353873dab06d5d06c31431d2559210cc not found: ID does not exist" containerID="97f29da7df242a4b2f87af0838c1102f353873dab06d5d06c31431d2559210cc"
Oct 30 01:01:23.236991 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.236923    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97f29da7df242a4b2f87af0838c1102f353873dab06d5d06c31431d2559210cc"} err="failed to get container status \"97f29da7df242a4b2f87af0838c1102f353873dab06d5d06c31431d2559210cc\": rpc error: code = NotFound desc = could not find container \"97f29da7df242a4b2f87af0838c1102f353873dab06d5d06c31431d2559210cc\": container with ID starting with 97f29da7df242a4b2f87af0838c1102f353873dab06d5d06c31431d2559210cc not found: ID does not exist"
Oct 30 01:01:23.236991 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.236942    6099 scope.go:122] "RemoveContainer" containerID="1652164599eab84070cc7219bc6df07fdc449c1c9cfa7661091ff2acc9113ac7"
Oct 30 01:01:23.237230 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:01:23.237212    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1652164599eab84070cc7219bc6df07fdc449c1c9cfa7661091ff2acc9113ac7\": container with ID starting with 1652164599eab84070cc7219bc6df07fdc449c1c9cfa7661091ff2acc9113ac7 not found: ID does not exist" containerID="1652164599eab84070cc7219bc6df07fdc449c1c9cfa7661091ff2acc9113ac7"
Oct 30 01:01:23.237281 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.237236    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1652164599eab84070cc7219bc6df07fdc449c1c9cfa7661091ff2acc9113ac7"} err="failed to get container status \"1652164599eab84070cc7219bc6df07fdc449c1c9cfa7661091ff2acc9113ac7\": rpc error: code = NotFound desc = could not find container \"1652164599eab84070cc7219bc6df07fdc449c1c9cfa7661091ff2acc9113ac7\": container with ID starting with 1652164599eab84070cc7219bc6df07fdc449c1c9cfa7661091ff2acc9113ac7 not found: ID does not exist"
Oct 30 01:01:23.290727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.290303    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-podinfo\") pod \"90b346c4-cc85-4fd7-9578-0533e8e902a9\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") "
Oct 30 01:01:23.290727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.290424    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90b346c4-cc85-4fd7-9578-0533e8e902a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b346c4-cc85-4fd7-9578-0533e8e902a9-istiod-ca-cert\") pod \"90b346c4-cc85-4fd7-9578-0533e8e902a9\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") "
Oct 30 01:01:23.290727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.290464    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90b346c4-cc85-4fd7-9578-0533e8e902a9-kube-api-access-94blq\" (UniqueName: \"kubernetes.io/projected/90b346c4-cc85-4fd7-9578-0533e8e902a9-kube-api-access-94blq\") pod \"90b346c4-cc85-4fd7-9578-0533e8e902a9\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") "
Oct 30 01:01:23.290727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.290514    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-workload-socket\") pod \"90b346c4-cc85-4fd7-9578-0533e8e902a9\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") "
Oct 30 01:01:23.290727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.290549    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-envoy\") pod \"90b346c4-cc85-4fd7-9578-0533e8e902a9\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") "
Oct 30 01:01:23.290727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.290735    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-credential-socket\") pod \"90b346c4-cc85-4fd7-9578-0533e8e902a9\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") "
Oct 30 01:01:23.291224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.290807    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-data\") pod \"90b346c4-cc85-4fd7-9578-0533e8e902a9\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") "
Oct 30 01:01:23.291224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.290851    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-workload-certs\") pod \"90b346c4-cc85-4fd7-9578-0533e8e902a9\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") "
Oct 30 01:01:23.291224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.290969    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-token\" (UniqueName: \"kubernetes.io/projected/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-token\") pod \"90b346c4-cc85-4fd7-9578-0533e8e902a9\" (UID: \"90b346c4-cc85-4fd7-9578-0533e8e902a9\") "
Oct 30 01:01:23.297789 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.297689    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b346c4-cc85-4fd7-9578-0533e8e902a9-istiod-ca-cert" pod "90b346c4-cc85-4fd7-9578-0533e8e902a9" (UID: "90b346c4-cc85-4fd7-9578-0533e8e902a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:23.298507 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.298475    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-workload-socket" pod "90b346c4-cc85-4fd7-9578-0533e8e902a9" (UID: "90b346c4-cc85-4fd7-9578-0533e8e902a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:23.302798 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.300411    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-podinfo" pod "90b346c4-cc85-4fd7-9578-0533e8e902a9" (UID: "90b346c4-cc85-4fd7-9578-0533e8e902a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:01:23.302798 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.300568    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b346c4-cc85-4fd7-9578-0533e8e902a9-kube-api-access-94blq" pod "90b346c4-cc85-4fd7-9578-0533e8e902a9" (UID: "90b346c4-cc85-4fd7-9578-0533e8e902a9"). InnerVolumeSpecName "kube-api-access-94blq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:23.302798 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.302038    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-token" pod "90b346c4-cc85-4fd7-9578-0533e8e902a9" (UID: "90b346c4-cc85-4fd7-9578-0533e8e902a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:23.302798 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.302287    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-data" pod "90b346c4-cc85-4fd7-9578-0533e8e902a9" (UID: "90b346c4-cc85-4fd7-9578-0533e8e902a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:23.302798 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.302512    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-credential-socket" pod "90b346c4-cc85-4fd7-9578-0533e8e902a9" (UID: "90b346c4-cc85-4fd7-9578-0533e8e902a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:23.302798 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.302744    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-workload-certs" pod "90b346c4-cc85-4fd7-9578-0533e8e902a9" (UID: "90b346c4-cc85-4fd7-9578-0533e8e902a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:23.308143 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.308083    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-envoy" pod "90b346c4-cc85-4fd7-9578-0533e8e902a9" (UID: "90b346c4-cc85-4fd7-9578-0533e8e902a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:23.325995 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.325956    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:01:23.330024 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.327377    6099 patch_prober.go:29] interesting pod/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:01:23.330024 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.329970    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm" podUID="58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 30 01:01:23.396463 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.396345    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh" podStartSLOduration=2.029639169 podStartE2EDuration="2.396329261s" totalImagesPullingTime="366.690092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:23.393398986 +0000 UTC m=+1577.250306742" watchObservedRunningTime="2026-10-30 01:01:23.396329261 +0000 UTC m=+1577.253237015"
Oct 30 01:01:23.398562 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.398512    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:23.398725 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.398644    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b346c4-cc85-4fd7-9578-0533e8e902a9-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:23.398725 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.398661    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94blq\" (UniqueName: \"kubernetes.io/projected/90b346c4-cc85-4fd7-9578-0533e8e902a9-kube-api-access-94blq\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:23.398725 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.398671    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:23.398725 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.398680    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:23.398725 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.398688    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:23.398725 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.398697    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:23.398989 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.398741    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b346c4-cc85-4fd7-9578-0533e8e902a9-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:23.398989 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.398750    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b346c4-cc85-4fd7-9578-0533e8e902a9-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:23.601384 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.601346    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"]
Oct 30 01:01:23.606391 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:23.606359    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-601aea-openshift-default-8559b84459-xs799"]
Oct 30 01:01:24.291265 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:24.291227    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90b346c4-cc85-4fd7-9578-0533e8e902a9" path="/var/lib/kubelet/pods/90b346c4-cc85-4fd7-9578-0533e8e902a9/volumes"
Oct 30 01:01:24.326314 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:24.326280    6099 patch_prober.go:29] interesting pod/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:01:24.326467 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:24.326338    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm" podUID="58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 30 01:01:25.326194 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:25.326161    6099 patch_prober.go:29] interesting pod/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:01:25.326618 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:25.326223    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm" podUID="58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 30 01:01:26.337686 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.328490    6099 patch_prober.go:29] interesting pod/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:01:26.337686 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.328561    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm" podUID="58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 30 01:01:26.337686 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.336392    6099 generic.go:356] "Generic (PLEG): container finished" podID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" containerID="9a861b5c79b98757e8b1994dd7cde16f934741c92027facdec5bd733794b6b98" exitCode=0
Oct 30 01:01:26.381756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.380134    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp" event={"ID":"24e7a71a-4aa1-46d0-a903-a480cac72a2b","Type":"ContainerDied","Data":"9a861b5c79b98757e8b1994dd7cde16f934741c92027facdec5bd733794b6b98"}
Oct 30 01:01:26.381756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.380213    6099 scope.go:122] "RemoveContainer" containerID="bbacec21207034f88fa78397321b6ad51b8379ebc8ae7cd5c5698852bec3ab64"
Oct 30 01:01:26.450729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.443230    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 01:01:26.608727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.605259    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-envoy\") pod \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") "
Oct 30 01:01:26.608727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.605330    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e7a71a-4aa1-46d0-a903-a480cac72a2b-kube-api-access-vr5sk\" (UniqueName: \"kubernetes.io/projected/24e7a71a-4aa1-46d0-a903-a480cac72a2b-kube-api-access-vr5sk\") pod \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") "
Oct 30 01:01:26.608727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.605468    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istiod-ca-cert\") pod \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") "
Oct 30 01:01:26.608727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.605504    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-token\" (UniqueName: \"kubernetes.io/projected/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-token\") pod \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") "
Oct 30 01:01:26.608727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.605547    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-data\") pod \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") "
Oct 30 01:01:26.608727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.605795    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-credential-socket\") pod \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") "
Oct 30 01:01:26.608727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.605897    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-workload-socket\") pod \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") "
Oct 30 01:01:26.608727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.605932    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-workload-certs\") pod \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") "
Oct 30 01:01:26.608727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.605966    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-podinfo\") pod \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\" (UID: \"24e7a71a-4aa1-46d0-a903-a480cac72a2b\") "
Oct 30 01:01:26.621675 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.621024    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-credential-socket" pod "24e7a71a-4aa1-46d0-a903-a480cac72a2b" (UID: "24e7a71a-4aa1-46d0-a903-a480cac72a2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:26.621675 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.621240    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-data" pod "24e7a71a-4aa1-46d0-a903-a480cac72a2b" (UID: "24e7a71a-4aa1-46d0-a903-a480cac72a2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:26.621675 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.621418    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-workload-socket" pod "24e7a71a-4aa1-46d0-a903-a480cac72a2b" (UID: "24e7a71a-4aa1-46d0-a903-a480cac72a2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:26.621675 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.621594    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-workload-certs" pod "24e7a71a-4aa1-46d0-a903-a480cac72a2b" (UID: "24e7a71a-4aa1-46d0-a903-a480cac72a2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:26.624011 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.623942    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istiod-ca-cert" pod "24e7a71a-4aa1-46d0-a903-a480cac72a2b" (UID: "24e7a71a-4aa1-46d0-a903-a480cac72a2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:26.656943 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.655681    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-envoy" pod "24e7a71a-4aa1-46d0-a903-a480cac72a2b" (UID: "24e7a71a-4aa1-46d0-a903-a480cac72a2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:26.676764 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.662681    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"]
Oct 30 01:01:26.676764 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.662972    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn" podUID="29a75432-1847-4c51-96e8-ec245d55a103" containerName="istio-proxy" containerID="cri-o://06566bd0e84879d316ff1d2c44e663b6a494f1d3eafae93fe0c2e840b221dcf1" gracePeriod=30
Oct 30 01:01:26.732181 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.716745    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:26.732181 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.716781    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:26.732181 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.716807    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:26.732181 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.716821    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:26.732181 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.716835    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:26.732181 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.716846    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:26.737003 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.736970    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-token" pod "24e7a71a-4aa1-46d0-a903-a480cac72a2b" (UID: "24e7a71a-4aa1-46d0-a903-a480cac72a2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:26.737304 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.737278    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-podinfo" pod "24e7a71a-4aa1-46d0-a903-a480cac72a2b" (UID: "24e7a71a-4aa1-46d0-a903-a480cac72a2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:01:26.739736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.739685    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e7a71a-4aa1-46d0-a903-a480cac72a2b-kube-api-access-vr5sk" pod "24e7a71a-4aa1-46d0-a903-a480cac72a2b" (UID: "24e7a71a-4aa1-46d0-a903-a480cac72a2b"). InnerVolumeSpecName "kube-api-access-vr5sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:26.819863 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.819730    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:26.819863 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.819820    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr5sk\" (UniqueName: \"kubernetes.io/projected/24e7a71a-4aa1-46d0-a903-a480cac72a2b-kube-api-access-vr5sk\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:26.819863 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:26.819838    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e7a71a-4aa1-46d0-a903-a480cac72a2b-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:27.230864 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.230838    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 01:01:27.326955 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.326920    6099 patch_prober.go:29] interesting pod/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:01:27.327242 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.327207    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm" podUID="58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 30 01:01:27.331475 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.331446    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istiod-ca-cert\") pod \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") "
Oct 30 01:01:27.331572 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.331500    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-kube-api-access-7tzd7\" (UniqueName: \"kubernetes.io/projected/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-kube-api-access-7tzd7\") pod \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") "
Oct 30 01:01:27.331653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.331638    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-data\") pod \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") "
Oct 30 01:01:27.331693 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.331679    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-workload-certs\") pod \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") "
Oct 30 01:01:27.331781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.331766    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-credential-socket\") pod \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") "
Oct 30 01:01:27.331906 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.331890    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-podinfo\") pod \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") "
Oct 30 01:01:27.331966 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.331952    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-workload-socket\") pod \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") "
Oct 30 01:01:27.332002 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.331973    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-data" pod "90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" (UID: "90c1fcbd-ab57-49f6-a3e5-92eb3a95a999"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:27.332038 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.331998    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-envoy\") pod \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") "
Oct 30 01:01:27.332135 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.332121    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-token\" (UniqueName: \"kubernetes.io/projected/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-token\") pod \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\" (UID: \"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999\") "
Oct 30 01:01:27.332205 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.332185    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-credential-socket" pod "90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" (UID: "90c1fcbd-ab57-49f6-a3e5-92eb3a95a999"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:27.332269 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.332249    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istiod-ca-cert" pod "90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" (UID: "90c1fcbd-ab57-49f6-a3e5-92eb3a95a999"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:27.332411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.332373    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-workload-certs" pod "90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" (UID: "90c1fcbd-ab57-49f6-a3e5-92eb3a95a999"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:27.332569 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.332549    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-workload-socket" pod "90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" (UID: "90c1fcbd-ab57-49f6-a3e5-92eb3a95a999"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:27.332963 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.332942    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:27.333066 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.332966    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:27.333066 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.332980    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:27.333066 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.332993    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:27.333066 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.333008    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:27.334174 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.334157    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-kube-api-access-7tzd7" pod "90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" (UID: "90c1fcbd-ab57-49f6-a3e5-92eb3a95a999"). InnerVolumeSpecName "kube-api-access-7tzd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:27.337229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.337187    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-token" pod "90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" (UID: "90c1fcbd-ab57-49f6-a3e5-92eb3a95a999"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:27.337229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.337192    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-podinfo" pod "90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" (UID: "90c1fcbd-ab57-49f6-a3e5-92eb3a95a999"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:01:27.337613 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.337593    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-envoy" pod "90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" (UID: "90c1fcbd-ab57-49f6-a3e5-92eb3a95a999"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:27.354744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.353834    6099 generic.go:356] "Generic (PLEG): container finished" podID="90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" containerID="00485a7d725182467e342da4f7072ea674d22ace36ac7e5c20acfe157bc6b74f" exitCode=0
Oct 30 01:01:27.354744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.353937    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl" event={"ID":"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999","Type":"ContainerDied","Data":"00485a7d725182467e342da4f7072ea674d22ace36ac7e5c20acfe157bc6b74f"}
Oct 30 01:01:27.354744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.353979    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl" event={"ID":"90c1fcbd-ab57-49f6-a3e5-92eb3a95a999","Type":"ContainerDied","Data":"35d09ad1c61512e0da21d29f17cd9c696801398a17956b5de309369190f381e8"}
Oct 30 01:01:27.354744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.354005    6099 scope.go:122] "RemoveContainer" containerID="00485a7d725182467e342da4f7072ea674d22ace36ac7e5c20acfe157bc6b74f"
Oct 30 01:01:27.354744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.354098    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"
Oct 30 01:01:27.362595 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.362507    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp" event={"ID":"24e7a71a-4aa1-46d0-a903-a480cac72a2b","Type":"ContainerDied","Data":"7a30f2210cdb48250fb7fd4b89ccc8639f3ddce42851965a0aa8a0bddbb7bb2b"}
Oct 30 01:01:27.362595 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.362589    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"
Oct 30 01:01:27.409034 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.408656    6099 scope.go:122] "RemoveContainer" containerID="00485a7d725182467e342da4f7072ea674d22ace36ac7e5c20acfe157bc6b74f"
Oct 30 01:01:27.409034 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:01:27.409103    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00485a7d725182467e342da4f7072ea674d22ace36ac7e5c20acfe157bc6b74f\": container with ID starting with 00485a7d725182467e342da4f7072ea674d22ace36ac7e5c20acfe157bc6b74f not found: ID does not exist" containerID="00485a7d725182467e342da4f7072ea674d22ace36ac7e5c20acfe157bc6b74f"
Oct 30 01:01:27.409463 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.409138    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00485a7d725182467e342da4f7072ea674d22ace36ac7e5c20acfe157bc6b74f"} err="failed to get container status \"00485a7d725182467e342da4f7072ea674d22ace36ac7e5c20acfe157bc6b74f\": rpc error: code = NotFound desc = could not find container \"00485a7d725182467e342da4f7072ea674d22ace36ac7e5c20acfe157bc6b74f\": container with ID starting with 00485a7d725182467e342da4f7072ea674d22ace36ac7e5c20acfe157bc6b74f not found: ID does not exist"
Oct 30 01:01:27.409463 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.409165    6099 scope.go:122] "RemoveContainer" containerID="9a861b5c79b98757e8b1994dd7cde16f934741c92027facdec5bd733794b6b98"
Oct 30 01:01:27.434984 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.434957    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:27.434984 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.434983    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tzd7\" (UniqueName: \"kubernetes.io/projected/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-kube-api-access-7tzd7\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:27.435201 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.434992    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:27.435201 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.435001    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:27.507727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.506791    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"]
Oct 30 01:01:27.513729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.512986    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-kd9xl"]
Oct 30 01:01:27.557082 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.554782    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"]
Oct 30 01:01:27.559771 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.559744    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-7lxvp"]
Oct 30 01:01:27.994730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.993690    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"]
Oct 30 01:01:27.995736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.995113    6099 cpu_manager.go:414] "containerMap: removing container" podUID="90b346c4-cc85-4fd7-9578-0533e8e902a9" containerName="istio-proxy"
Oct 30 01:01:27.995736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.995157    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90b346c4-cc85-4fd7-9578-0533e8e902a9" containerName="istio-proxy"
Oct 30 01:01:27.995736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.995223    6099 cpu_manager.go:414] "containerMap: removing container" podUID="90b346c4-cc85-4fd7-9578-0533e8e902a9" containerName="istio-proxy"
Oct 30 01:01:27.995736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.995233    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90b346c4-cc85-4fd7-9578-0533e8e902a9" containerName="istio-proxy"
Oct 30 01:01:27.995736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.995270    6099 cpu_manager.go:414] "containerMap: removing container" podUID="90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" containerName="istio-proxy"
Oct 30 01:01:27.995736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.995278    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" containerName="istio-proxy"
Oct 30 01:01:27.995736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.995294    6099 cpu_manager.go:414] "containerMap: removing container" podUID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" containerName="istio-proxy"
Oct 30 01:01:27.995736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.995302    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" containerName="istio-proxy"
Oct 30 01:01:27.995736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.995323    6099 cpu_manager.go:414] "containerMap: removing container" podUID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" containerName="istio-proxy"
Oct 30 01:01:27.995736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.995343    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" containerName="istio-proxy"
Oct 30 01:01:27.995736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.995527    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="90b346c4-cc85-4fd7-9578-0533e8e902a9" containerName="istio-proxy"
Oct 30 01:01:27.995736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.995558    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" containerName="istio-proxy"
Oct 30 01:01:27.995736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.995569    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" containerName="istio-proxy"
Oct 30 01:01:27.995736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:27.995579    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" containerName="istio-proxy"
Oct 30 01:01:28.005653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.005628    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"]
Oct 30 01:01:28.005841 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.005794    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.037973 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.037953    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-cf3466-openshift-default-dockercfg-z4nfv\""
Oct 30 01:01:28.053937 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.053915    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-credential-socket\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.054042 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.053952    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-data\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.054125 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.054050    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-podinfo\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.054200 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.054137    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-envoy\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.054200 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.054183    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-token\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.054307 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.054225    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istiod-ca-cert\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.054366 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.054311    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwhb7\" (UniqueName: \"kubernetes.io/projected/502b2376-6a3e-4c6d-9cab-aa704bcd3053-kube-api-access-nwhb7\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.054411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.054368    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-workload-socket\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.054449 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.054430    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-workload-certs\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.169732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.164746    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-podinfo\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.169732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.164815    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-envoy\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.169732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.164841    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-token\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.169732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.164879    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istiod-ca-cert\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.169732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.164925    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwhb7\" (UniqueName: \"kubernetes.io/projected/502b2376-6a3e-4c6d-9cab-aa704bcd3053-kube-api-access-nwhb7\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.169732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.164973    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-workload-socket\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.169732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.164999    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-workload-certs\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.169732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.165238    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-credential-socket\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.169732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.165270    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-data\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.169732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.165786    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-data\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.169732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.166073    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-workload-socket\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.175730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.173987    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-workload-certs\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.175730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.174251    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-credential-socket\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.198731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.187597    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istiod-ca-cert\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.223103 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.222365    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-podinfo\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.223940 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.223917    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-envoy\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.231217 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.231171    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-token\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.231806 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.231777    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwhb7\" (UniqueName: \"kubernetes.io/projected/502b2376-6a3e-4c6d-9cab-aa704bcd3053-kube-api-access-nwhb7\") pod \"e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") " pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.330728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.330179    6099 patch_prober.go:29] interesting pod/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 30 01:01:28.330728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.330242    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm" podUID="58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 30 01:01:28.331012 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.330880    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:28.364395 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.364353    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24e7a71a-4aa1-46d0-a903-a480cac72a2b" path="/var/lib/kubelet/pods/24e7a71a-4aa1-46d0-a903-a480cac72a2b/volumes"
Oct 30 01:01:28.378730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.372273    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90c1fcbd-ab57-49f6-a3e5-92eb3a95a999" path="/var/lib/kubelet/pods/90c1fcbd-ab57-49f6-a3e5-92eb3a95a999/volumes"
Oct 30 01:01:28.773424 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:01:28.773388    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502b2376_6a3e_4c6d_9cab_aa704bcd3053.slice/crio-ae6857c3d8fa9c8bc6d5e0abfc30af759248154a8ba366c9678319999128b73c WatchSource:0}: Error finding container ae6857c3d8fa9c8bc6d5e0abfc30af759248154a8ba366c9678319999128b73c: Status 404 returned error can't find the container with id ae6857c3d8fa9c8bc6d5e0abfc30af759248154a8ba366c9678319999128b73c
Oct 30 01:01:28.782511 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.782473    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"]
Oct 30 01:01:28.786302 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.786251    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:01:28.786439 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.786398    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:01:28.786515 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:28.786490    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:01:29.234933 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:29.234862    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"
Oct 30 01:01:29.332010 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:29.331978    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:01:29.332408 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:29.332384    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:01:29.333207 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:29.333185    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:01:29.422770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:29.422738    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c" event={"ID":"502b2376-6a3e-4c6d-9cab-aa704bcd3053","Type":"ContainerStarted","Data":"846f119fc4eeffdcc2136c24c8139230a91c411b2c2fdad8a5891e0877a8fabb"}
Oct 30 01:01:29.423226 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:29.422790    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c" event={"ID":"502b2376-6a3e-4c6d-9cab-aa704bcd3053","Type":"ContainerStarted","Data":"ae6857c3d8fa9c8bc6d5e0abfc30af759248154a8ba366c9678319999128b73c"}
Oct 30 01:01:29.519132 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:29.519039    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c" podStartSLOduration=2.519025005 podStartE2EDuration="2.519025005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:29.513942006 +0000 UTC m=+1583.370849775" watchObservedRunningTime="2026-10-30 01:01:29.519025005 +0000 UTC m=+1583.375932816"
Oct 30 01:01:29.756212 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:29.756173    6099 patch_prober.go:29] interesting pod/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:01:29.756404 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:29.756242    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn" podUID="29a75432-1847-4c51-96e8-ec245d55a103" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:01:30.332899 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:30.332859    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:30.339356 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:30.339331    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:30.433860 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:30.432465    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:30.433860 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:30.433500    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:01:31.965212 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:31.965186    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 01:01:32.062814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.060333    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a75432-1847-4c51-96e8-ec245d55a103-istio-token\" (UniqueName: \"kubernetes.io/projected/29a75432-1847-4c51-96e8-ec245d55a103-istio-token\") pod \"29a75432-1847-4c51-96e8-ec245d55a103\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") "
Oct 30 01:01:32.062814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.060419    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-istio-data\") pod \"29a75432-1847-4c51-96e8-ec245d55a103\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") "
Oct 30 01:01:32.062814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.060502    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a75432-1847-4c51-96e8-ec245d55a103-kube-api-access-mmcxz\" (UniqueName: \"kubernetes.io/projected/29a75432-1847-4c51-96e8-ec245d55a103-kube-api-access-mmcxz\") pod \"29a75432-1847-4c51-96e8-ec245d55a103\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") "
Oct 30 01:01:32.062814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.060593    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29a75432-1847-4c51-96e8-ec245d55a103-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a75432-1847-4c51-96e8-ec245d55a103-istio-podinfo\") pod \"29a75432-1847-4c51-96e8-ec245d55a103\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") "
Oct 30 01:01:32.062814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.060661    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-workload-certs\") pod \"29a75432-1847-4c51-96e8-ec245d55a103\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") "
Oct 30 01:01:32.062814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.060691    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-istio-envoy\") pod \"29a75432-1847-4c51-96e8-ec245d55a103\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") "
Oct 30 01:01:32.062814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.060796    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-credential-socket\") pod \"29a75432-1847-4c51-96e8-ec245d55a103\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") "
Oct 30 01:01:32.062814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.060904    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-workload-socket\") pod \"29a75432-1847-4c51-96e8-ec245d55a103\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") "
Oct 30 01:01:32.062814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.060941    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29a75432-1847-4c51-96e8-ec245d55a103-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a75432-1847-4c51-96e8-ec245d55a103-istiod-ca-cert\") pod \"29a75432-1847-4c51-96e8-ec245d55a103\" (UID: \"29a75432-1847-4c51-96e8-ec245d55a103\") "
Oct 30 01:01:32.083692 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.083646    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a75432-1847-4c51-96e8-ec245d55a103-istiod-ca-cert" pod "29a75432-1847-4c51-96e8-ec245d55a103" (UID: "29a75432-1847-4c51-96e8-ec245d55a103"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:32.084060 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.084021    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-istio-data" pod "29a75432-1847-4c51-96e8-ec245d55a103" (UID: "29a75432-1847-4c51-96e8-ec245d55a103"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:32.087299 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.087266    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-workload-certs" pod "29a75432-1847-4c51-96e8-ec245d55a103" (UID: "29a75432-1847-4c51-96e8-ec245d55a103"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:32.087509 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.087486    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-credential-socket" pod "29a75432-1847-4c51-96e8-ec245d55a103" (UID: "29a75432-1847-4c51-96e8-ec245d55a103"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:32.087694 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.087664    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-workload-socket" pod "29a75432-1847-4c51-96e8-ec245d55a103" (UID: "29a75432-1847-4c51-96e8-ec245d55a103"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:32.088166 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.088093    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29a75432-1847-4c51-96e8-ec245d55a103-istio-podinfo" pod "29a75432-1847-4c51-96e8-ec245d55a103" (UID: "29a75432-1847-4c51-96e8-ec245d55a103"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:01:32.088691 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.088626    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a75432-1847-4c51-96e8-ec245d55a103-istio-token" pod "29a75432-1847-4c51-96e8-ec245d55a103" (UID: "29a75432-1847-4c51-96e8-ec245d55a103"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:32.088960 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.088898    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-istio-envoy" pod "29a75432-1847-4c51-96e8-ec245d55a103" (UID: "29a75432-1847-4c51-96e8-ec245d55a103"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:32.088960 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.088905    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a75432-1847-4c51-96e8-ec245d55a103-kube-api-access-mmcxz" pod "29a75432-1847-4c51-96e8-ec245d55a103" (UID: "29a75432-1847-4c51-96e8-ec245d55a103"). InnerVolumeSpecName "kube-api-access-mmcxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:32.167690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.165739    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a75432-1847-4c51-96e8-ec245d55a103-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:32.167690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.165784    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:32.167690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.165799    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmcxz\" (UniqueName: \"kubernetes.io/projected/29a75432-1847-4c51-96e8-ec245d55a103-kube-api-access-mmcxz\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:32.167690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.165814    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a75432-1847-4c51-96e8-ec245d55a103-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:32.167690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.165827    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:32.167690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.165845    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:32.167690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.165857    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:32.167690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.165870    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a75432-1847-4c51-96e8-ec245d55a103-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:32.167690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.165883    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a75432-1847-4c51-96e8-ec245d55a103-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:01:32.452052 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.452010    6099 generic.go:356] "Generic (PLEG): container finished" podID="29a75432-1847-4c51-96e8-ec245d55a103" containerID="06566bd0e84879d316ff1d2c44e663b6a494f1d3eafae93fe0c2e840b221dcf1" exitCode=0
Oct 30 01:01:32.452267 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.452098    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"
Oct 30 01:01:32.452267 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.452178    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn" event={"ID":"29a75432-1847-4c51-96e8-ec245d55a103","Type":"ContainerDied","Data":"06566bd0e84879d316ff1d2c44e663b6a494f1d3eafae93fe0c2e840b221dcf1"}
Oct 30 01:01:32.452267 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.452244    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn" event={"ID":"29a75432-1847-4c51-96e8-ec245d55a103","Type":"ContainerDied","Data":"61732f2435524e66db469f5bbe85ea8504674e0d981af19ad346c6fc9f903e38"}
Oct 30 01:01:32.452415 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.452269    6099 scope.go:122] "RemoveContainer" containerID="06566bd0e84879d316ff1d2c44e663b6a494f1d3eafae93fe0c2e840b221dcf1"
Oct 30 01:01:32.465423 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.465401    6099 scope.go:122] "RemoveContainer" containerID="784d620e38d6a18ac1b3cc95e1db0dada1e1913cc211c848857531f968a98119"
Oct 30 01:01:32.476590 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.476570    6099 scope.go:122] "RemoveContainer" containerID="06566bd0e84879d316ff1d2c44e663b6a494f1d3eafae93fe0c2e840b221dcf1"
Oct 30 01:01:32.476928 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:01:32.476892    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06566bd0e84879d316ff1d2c44e663b6a494f1d3eafae93fe0c2e840b221dcf1\": container with ID starting with 06566bd0e84879d316ff1d2c44e663b6a494f1d3eafae93fe0c2e840b221dcf1 not found: ID does not exist" containerID="06566bd0e84879d316ff1d2c44e663b6a494f1d3eafae93fe0c2e840b221dcf1"
Oct 30 01:01:32.477099 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.476935    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06566bd0e84879d316ff1d2c44e663b6a494f1d3eafae93fe0c2e840b221dcf1"} err="failed to get container status \"06566bd0e84879d316ff1d2c44e663b6a494f1d3eafae93fe0c2e840b221dcf1\": rpc error: code = NotFound desc = could not find container \"06566bd0e84879d316ff1d2c44e663b6a494f1d3eafae93fe0c2e840b221dcf1\": container with ID starting with 06566bd0e84879d316ff1d2c44e663b6a494f1d3eafae93fe0c2e840b221dcf1 not found: ID does not exist"
Oct 30 01:01:32.477099 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.476959    6099 scope.go:122] "RemoveContainer" containerID="784d620e38d6a18ac1b3cc95e1db0dada1e1913cc211c848857531f968a98119"
Oct 30 01:01:32.477370 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:01:32.477287    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"784d620e38d6a18ac1b3cc95e1db0dada1e1913cc211c848857531f968a98119\": container with ID starting with 784d620e38d6a18ac1b3cc95e1db0dada1e1913cc211c848857531f968a98119 not found: ID does not exist" containerID="784d620e38d6a18ac1b3cc95e1db0dada1e1913cc211c848857531f968a98119"
Oct 30 01:01:32.477370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.477311    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784d620e38d6a18ac1b3cc95e1db0dada1e1913cc211c848857531f968a98119"} err="failed to get container status \"784d620e38d6a18ac1b3cc95e1db0dada1e1913cc211c848857531f968a98119\": rpc error: code = NotFound desc = could not find container \"784d620e38d6a18ac1b3cc95e1db0dada1e1913cc211c848857531f968a98119\": container with ID starting with 784d620e38d6a18ac1b3cc95e1db0dada1e1913cc211c848857531f968a98119 not found: ID does not exist"
Oct 30 01:01:32.532312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.532210    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"]
Oct 30 01:01:32.543522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:32.543496    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a32412-openshift-default-7d4f8c46bf-2z9gn"]
Oct 30 01:01:33.502260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.502224    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"]
Oct 30 01:01:33.503182 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.503158    6099 cpu_manager.go:414] "containerMap: removing container" podUID="29a75432-1847-4c51-96e8-ec245d55a103" containerName="istio-proxy"
Oct 30 01:01:33.503300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.503184    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a75432-1847-4c51-96e8-ec245d55a103" containerName="istio-proxy"
Oct 30 01:01:33.503300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.503274    6099 cpu_manager.go:414] "containerMap: removing container" podUID="29a75432-1847-4c51-96e8-ec245d55a103" containerName="istio-proxy"
Oct 30 01:01:33.503300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.503283    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a75432-1847-4c51-96e8-ec245d55a103" containerName="istio-proxy"
Oct 30 01:01:33.503487 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.503472    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a75432-1847-4c51-96e8-ec245d55a103" containerName="istio-proxy"
Oct 30 01:01:33.503547 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.503498    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="90b346c4-cc85-4fd7-9578-0533e8e902a9" containerName="istio-proxy"
Oct 30 01:01:33.503547 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.503513    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a75432-1847-4c51-96e8-ec245d55a103" containerName="istio-proxy"
Oct 30 01:01:33.508763 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.508739    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"
Oct 30 01:01:33.528205 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.528173    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"]
Oct 30 01:01:33.534427 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.534398    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2a8b26-serving-cert\""
Oct 30 01:01:33.692168 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.692136    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/261c5da2-a187-4ad5-876f-35db0c631a06-maas-api-tls\") pod \"maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46\" (UID: \"261c5da2-a187-4ad5-876f-35db0c631a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"
Oct 30 01:01:33.692168 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.692169    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4z2f\" (UniqueName: \"kubernetes.io/projected/261c5da2-a187-4ad5-876f-35db0c631a06-kube-api-access-t4z2f\") pod \"maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46\" (UID: \"261c5da2-a187-4ad5-876f-35db0c631a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"
Oct 30 01:01:33.692404 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.692254    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/261c5da2-a187-4ad5-876f-35db0c631a06-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46\" (UID: \"261c5da2-a187-4ad5-876f-35db0c631a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"
Oct 30 01:01:33.692450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.692418    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/261c5da2-a187-4ad5-876f-35db0c631a06-metrics-certs\") pod \"maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46\" (UID: \"261c5da2-a187-4ad5-876f-35db0c631a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"
Oct 30 01:01:33.794366 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.794331    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/261c5da2-a187-4ad5-876f-35db0c631a06-metrics-certs\") pod \"maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46\" (UID: \"261c5da2-a187-4ad5-876f-35db0c631a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"
Oct 30 01:01:33.794529 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.794500    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/261c5da2-a187-4ad5-876f-35db0c631a06-maas-api-tls\") pod \"maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46\" (UID: \"261c5da2-a187-4ad5-876f-35db0c631a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"
Oct 30 01:01:33.794529 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.794518    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4z2f\" (UniqueName: \"kubernetes.io/projected/261c5da2-a187-4ad5-876f-35db0c631a06-kube-api-access-t4z2f\") pod \"maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46\" (UID: \"261c5da2-a187-4ad5-876f-35db0c631a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"
Oct 30 01:01:33.794604 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.794552    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/261c5da2-a187-4ad5-876f-35db0c631a06-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46\" (UID: \"261c5da2-a187-4ad5-876f-35db0c631a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"
Oct 30 01:01:33.795227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.795199    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/261c5da2-a187-4ad5-876f-35db0c631a06-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46\" (UID: \"261c5da2-a187-4ad5-876f-35db0c631a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"
Oct 30 01:01:33.797204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.797177    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/261c5da2-a187-4ad5-876f-35db0c631a06-maas-api-tls\") pod \"maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46\" (UID: \"261c5da2-a187-4ad5-876f-35db0c631a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"
Oct 30 01:01:33.797353 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.797245    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/261c5da2-a187-4ad5-876f-35db0c631a06-metrics-certs\") pod \"maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46\" (UID: \"261c5da2-a187-4ad5-876f-35db0c631a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"
Oct 30 01:01:33.818608 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.818570    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4z2f\" (UniqueName: \"kubernetes.io/projected/261c5da2-a187-4ad5-876f-35db0c631a06-kube-api-access-t4z2f\") pod \"maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46\" (UID: \"261c5da2-a187-4ad5-876f-35db0c631a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"
Oct 30 01:01:33.823382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:33.823354    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"
Oct 30 01:01:34.143289 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:01:34.143248    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261c5da2_a187_4ad5_876f_35db0c631a06.slice/crio-838d80f565575b2dcadb5645b02e21fad0118659f8baed647af1614b87875a1a WatchSource:0}: Error finding container 838d80f565575b2dcadb5645b02e21fad0118659f8baed647af1614b87875a1a: Status 404 returned error can't find the container with id 838d80f565575b2dcadb5645b02e21fad0118659f8baed647af1614b87875a1a
Oct 30 01:01:34.144036 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:34.144013    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"]
Oct 30 01:01:34.263447 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:34.263413    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a75432-1847-4c51-96e8-ec245d55a103" path="/var/lib/kubelet/pods/29a75432-1847-4c51-96e8-ec245d55a103/volumes"
Oct 30 01:01:34.474561 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:34.474485    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46" event={"ID":"261c5da2-a187-4ad5-876f-35db0c631a06","Type":"ContainerStarted","Data":"838d80f565575b2dcadb5645b02e21fad0118659f8baed647af1614b87875a1a"}
Oct 30 01:01:34.693154 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:34.693124    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf"]
Oct 30 01:01:34.699225 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:34.699206    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf"
Oct 30 01:01:34.716009 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:34.715977    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf\" (UID: \"f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf"
Oct 30 01:01:34.716139 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:34.716069    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt82p\" (UniqueName: \"kubernetes.io/projected/f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2-kube-api-access-dt82p\") pod \"payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf\" (UID: \"f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf"
Oct 30 01:01:34.720340 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:34.720313    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf"]
Oct 30 01:01:34.726884 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:34.726826    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-cf3466\""
Oct 30 01:01:34.810729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:34.802031    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-cf3466-dockercfg-68csh\""
Oct 30 01:01:34.858767 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:34.857162    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf\" (UID: \"f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf"
Oct 30 01:01:34.858767 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:34.857275    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt82p\" (UniqueName: \"kubernetes.io/projected/f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2-kube-api-access-dt82p\") pod \"payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf\" (UID: \"f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf"
Oct 30 01:01:34.859030 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:34.858886    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf\" (UID: \"f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf"
Oct 30 01:01:34.953402 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:34.953147    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt82p\" (UniqueName: \"kubernetes.io/projected/f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2-kube-api-access-dt82p\") pod \"payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf\" (UID: \"f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf"
Oct 30 01:01:35.021256 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:35.019691    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf"
Oct 30 01:01:35.349954 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:01:35.349912    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10e5a00_8ac3_4382_948f_b9fe4b3ea9c2.slice/crio-0428a29125c6f8f0787e4aa17803de5c5649d3b92789ef05b886d8fd54e16bf8 WatchSource:0}: Error finding container 0428a29125c6f8f0787e4aa17803de5c5649d3b92789ef05b886d8fd54e16bf8: Status 404 returned error can't find the container with id 0428a29125c6f8f0787e4aa17803de5c5649d3b92789ef05b886d8fd54e16bf8
Oct 30 01:01:35.351364 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:35.351340    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf"]
Oct 30 01:01:35.484773 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:35.484695    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf" event={"ID":"f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2","Type":"ContainerStarted","Data":"0428a29125c6f8f0787e4aa17803de5c5649d3b92789ef05b886d8fd54e16bf8"}
Oct 30 01:01:35.486315 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:35.486294    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46" event={"ID":"261c5da2-a187-4ad5-876f-35db0c631a06","Type":"ContainerStarted","Data":"8266ea777fc0426ee26eada0ed52cf4bb6ca09c71d990ba92b0e7986a671723f"}
Oct 30 01:01:35.486434 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:35.486351    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"
Oct 30 01:01:35.529392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:35.529342    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46" podStartSLOduration=2.048228215 podStartE2EDuration="2.529329988s" totalImagesPullingTime="481.101773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:35.527693145 +0000 UTC m=+1589.384600902" watchObservedRunningTime="2026-10-30 01:01:35.529329988 +0000 UTC m=+1589.386237742"
Oct 30 01:01:36.536177 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:36.536136    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf" event={"ID":"f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2","Type":"ContainerStarted","Data":"8d4006030d0fe5314aa32312d39dc15a8fb2a66c67de51fb813e56cf66944ce3"}
Oct 30 01:01:36.536687 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:36.536226    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf"
Oct 30 01:01:36.578246 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:36.578194    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf" podStartSLOduration=2.578178422 podStartE2EDuration="2.578178422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:36.574964009 +0000 UTC m=+1590.431871777" watchObservedRunningTime="2026-10-30 01:01:36.578178422 +0000 UTC m=+1590.435086195"
Oct 30 01:01:41.541834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:41.541800    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"
Oct 30 01:01:46.633245 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:01:46.633203    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:01:47.547967 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:47.547926    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf"
Oct 30 01:01:52.298301 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:01:52.298263    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:02:04.106929 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:04.106862    6099 patch_prober.go:29] interesting pod/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:02:04.107432 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:04.106943    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf" podUID="c9a34143-9d52-4d6a-99d3-05d6501f1bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:04.765873 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:04.765837    6099 patch_prober.go:29] interesting pod/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:02:04.766060 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:04.765930    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44" podUID="7c6446b4-b8c2-42b3-b449-2ae9141a19dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:06.121961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:06.121870    6099 generic.go:356] "Generic (PLEG): container finished" podID="f85bad1f-83ed-42b6-b88a-c1388b5480b2" containerID="a1f83ceeed3b9e9fcedc6c53c3b8dea1ca41a624a2b71466bf8ce695c80c77fa" exitCode=0
Oct 30 01:02:06.122471 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:06.122064    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48" event={"ID":"f85bad1f-83ed-42b6-b88a-c1388b5480b2","Type":"ContainerDied","Data":"a1f83ceeed3b9e9fcedc6c53c3b8dea1ca41a624a2b71466bf8ce695c80c77fa"}
Oct 30 01:02:06.122557 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:06.122506    6099 scope.go:122] "RemoveContainer" containerID="a1f83ceeed3b9e9fcedc6c53c3b8dea1ca41a624a2b71466bf8ce695c80c77fa"
Oct 30 01:02:06.125008 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:06.124963    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:02:06.125191 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:06.125040    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:02:06.125191 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:06.125071    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:02:06.793030 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:06.792983    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:02:07.132752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:07.132643    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48" event={"ID":"f85bad1f-83ed-42b6-b88a-c1388b5480b2","Type":"ContainerStarted","Data":"16c29f97a01515549fb10b641f313d731bf64b45a0ad1f8b0a3cd67002e2de39"}
Oct 30 01:02:07.609663 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:07.609624    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:02:07.610727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:07.610687    6099 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 01:02:07.610791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:07.610766    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48" podUID="f85bad1f-83ed-42b6-b88a-c1388b5480b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 01:02:08.144862 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:08.144775    6099 generic.go:356] "Generic (PLEG): container finished" podID="c9a34143-9d52-4d6a-99d3-05d6501f1bb7" containerID="86971e1ba50bbb98ffa4e637644d94ed90d0edbc81b42f053f4584241f82a27a" exitCode=0
Oct 30 01:02:08.145280 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:08.144854    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf" event={"ID":"c9a34143-9d52-4d6a-99d3-05d6501f1bb7","Type":"ContainerDied","Data":"86971e1ba50bbb98ffa4e637644d94ed90d0edbc81b42f053f4584241f82a27a"}
Oct 30 01:02:08.145396 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:08.145376    6099 scope.go:122] "RemoveContainer" containerID="86971e1ba50bbb98ffa4e637644d94ed90d0edbc81b42f053f4584241f82a27a"
Oct 30 01:02:08.146918 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:08.146894    6099 generic.go:356] "Generic (PLEG): container finished" podID="7c6446b4-b8c2-42b3-b449-2ae9141a19dc" containerID="734491f5f7fe82ea06f75bf7eee81e6b6d172c3a1961436b460e56a17d464e0b" exitCode=0
Oct 30 01:02:08.147051 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:08.146973    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44" event={"ID":"7c6446b4-b8c2-42b3-b449-2ae9141a19dc","Type":"ContainerDied","Data":"734491f5f7fe82ea06f75bf7eee81e6b6d172c3a1961436b460e56a17d464e0b"}
Oct 30 01:02:08.147462 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:08.147445    6099 scope.go:122] "RemoveContainer" containerID="734491f5f7fe82ea06f75bf7eee81e6b6d172c3a1961436b460e56a17d464e0b"
Oct 30 01:02:08.147569 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:08.147444    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:02:08.147569 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:08.147519    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:02:08.147569 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:08.147555    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:02:08.149257 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:08.149223    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:02:08.149376 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:08.149301    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:02:08.149376 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:08.149350    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:02:08.612094 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:08.611083    6099 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 01:02:08.612094 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:08.611151    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48" podUID="f85bad1f-83ed-42b6-b88a-c1388b5480b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 01:02:09.244248 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:09.244209    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf" event={"ID":"c9a34143-9d52-4d6a-99d3-05d6501f1bb7","Type":"ContainerStarted","Data":"a20a492200614723d9316ac3ac8826dc9488350f5a7c6baaa3db7da36b97f122"}
Oct 30 01:02:09.251109 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:09.249851    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44" event={"ID":"7c6446b4-b8c2-42b3-b449-2ae9141a19dc","Type":"ContainerStarted","Data":"52be99d2493f3c23a6f7538e8b985abe12d94d2aceccc62f1ed6e1e6713c4656"}
Oct 30 01:02:09.582731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:09.569133    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:02:09.582731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:09.581081    6099 patch_prober.go:29] interesting pod/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 30 01:02:09.582731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:09.581159    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44" podUID="7c6446b4-b8c2-42b3-b449-2ae9141a19dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 30 01:02:09.620729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:09.616872    6099 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 01:02:09.620729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:09.616952    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48" podUID="f85bad1f-83ed-42b6-b88a-c1388b5480b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 01:02:09.698230 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:09.698124    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:02:09.698899 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:09.698553    6099 patch_prober.go:29] interesting pod/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 30 01:02:09.698899 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:09.698611    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf" podUID="c9a34143-9d52-4d6a-99d3-05d6501f1bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 30 01:02:10.573227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:10.573091    6099 patch_prober.go:29] interesting pod/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 30 01:02:10.573227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:10.573151    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44" podUID="7c6446b4-b8c2-42b3-b449-2ae9141a19dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 30 01:02:10.609097 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:10.609061    6099 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 01:02:10.621535 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:10.621495    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48" podUID="f85bad1f-83ed-42b6-b88a-c1388b5480b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 01:02:10.700732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:10.698086    6099 patch_prober.go:29] interesting pod/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 30 01:02:10.700732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:10.698150    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf" podUID="c9a34143-9d52-4d6a-99d3-05d6501f1bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 30 01:02:11.580740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:11.575849    6099 patch_prober.go:29] interesting pod/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 30 01:02:11.580740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:11.575911    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44" podUID="7c6446b4-b8c2-42b3-b449-2ae9141a19dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 30 01:02:11.644655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:11.644617    6099 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 01:02:11.648993 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:11.648950    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48" podUID="f85bad1f-83ed-42b6-b88a-c1388b5480b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 01:02:11.704741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:11.702406    6099 patch_prober.go:29] interesting pod/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 30 01:02:11.704741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:11.702482    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf" podUID="c9a34143-9d52-4d6a-99d3-05d6501f1bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 30 01:02:12.583732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:12.567569    6099 patch_prober.go:29] interesting pod/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 30 01:02:12.583732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:12.567628    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44" podUID="7c6446b4-b8c2-42b3-b449-2ae9141a19dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 30 01:02:12.635741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:12.633817    6099 patch_prober.go:29] interesting pod/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 01:02:12.635741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:12.633878    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48" podUID="f85bad1f-83ed-42b6-b88a-c1388b5480b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 01:02:12.698733 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:12.698043    6099 patch_prober.go:29] interesting pod/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 30 01:02:12.698733 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:12.698102    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf" podUID="c9a34143-9d52-4d6a-99d3-05d6501f1bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 30 01:02:13.115537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:13.115503    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx"]
Oct 30 01:02:13.192802 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:13.190577    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6aa0d68-cv5lx"]
Oct 30 01:02:13.570743 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:13.569047    6099 patch_prober.go:29] interesting pod/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 30 01:02:13.570743 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:13.569107    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44" podUID="7c6446b4-b8c2-42b3-b449-2ae9141a19dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 30 01:02:13.629413 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:13.627140    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:02:13.629413 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:13.628783    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:02:13.631086 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:13.631061    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:02:13.708731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:13.708296    6099 patch_prober.go:29] interesting pod/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 30 01:02:13.708731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:13.708352    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf" podUID="c9a34143-9d52-4d6a-99d3-05d6501f1bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 30 01:02:14.264124 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:14.264087    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98809b25-de5f-4cb7-add8-6a0aa323aa9d" path="/var/lib/kubelet/pods/98809b25-de5f-4cb7-add8-6a0aa323aa9d/volumes"
Oct 30 01:02:14.582104 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:14.582069    6099 patch_prober.go:29] interesting pod/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 30 01:02:14.582412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:14.582385    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44" podUID="7c6446b4-b8c2-42b3-b449-2ae9141a19dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 30 01:02:14.699190 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:14.699156    6099 patch_prober.go:29] interesting pod/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 30 01:02:14.699815 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:14.699761    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf" podUID="c9a34143-9d52-4d6a-99d3-05d6501f1bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 30 01:02:14.829097 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:14.829052    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"]
Oct 30 01:02:14.838917 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:14.838659    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"]
Oct 30 01:02:14.838917 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:14.838809    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"
Oct 30 01:02:14.856360 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:14.856147    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-cf3466-serving-cert\""
Oct 30 01:02:14.931679 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:14.931641    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/571da8ac-1d91-472b-80a1-187c16a687e8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n\" (UID: \"571da8ac-1d91-472b-80a1-187c16a687e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"
Oct 30 01:02:14.931862 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:14.931849    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw4tm\" (UniqueName: \"kubernetes.io/projected/571da8ac-1d91-472b-80a1-187c16a687e8-kube-api-access-kw4tm\") pod \"maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n\" (UID: \"571da8ac-1d91-472b-80a1-187c16a687e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"
Oct 30 01:02:14.931963 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:14.931950    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/571da8ac-1d91-472b-80a1-187c16a687e8-maas-api-tls\") pod \"maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n\" (UID: \"571da8ac-1d91-472b-80a1-187c16a687e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"
Oct 30 01:02:14.932025 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:14.932012    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/571da8ac-1d91-472b-80a1-187c16a687e8-metrics-certs\") pod \"maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n\" (UID: \"571da8ac-1d91-472b-80a1-187c16a687e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"
Oct 30 01:02:15.036738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:15.035813    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/571da8ac-1d91-472b-80a1-187c16a687e8-maas-api-tls\") pod \"maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n\" (UID: \"571da8ac-1d91-472b-80a1-187c16a687e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"
Oct 30 01:02:15.036738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:15.035925    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/571da8ac-1d91-472b-80a1-187c16a687e8-metrics-certs\") pod \"maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n\" (UID: \"571da8ac-1d91-472b-80a1-187c16a687e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"
Oct 30 01:02:15.036738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:15.035961    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/571da8ac-1d91-472b-80a1-187c16a687e8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n\" (UID: \"571da8ac-1d91-472b-80a1-187c16a687e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"
Oct 30 01:02:15.036738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:15.036198    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw4tm\" (UniqueName: \"kubernetes.io/projected/571da8ac-1d91-472b-80a1-187c16a687e8-kube-api-access-kw4tm\") pod \"maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n\" (UID: \"571da8ac-1d91-472b-80a1-187c16a687e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"
Oct 30 01:02:15.055006 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:15.048863    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/571da8ac-1d91-472b-80a1-187c16a687e8-maas-api-tls\") pod \"maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n\" (UID: \"571da8ac-1d91-472b-80a1-187c16a687e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"
Oct 30 01:02:15.055006 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:15.049519    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/571da8ac-1d91-472b-80a1-187c16a687e8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n\" (UID: \"571da8ac-1d91-472b-80a1-187c16a687e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"
Oct 30 01:02:15.055006 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:15.054474    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/571da8ac-1d91-472b-80a1-187c16a687e8-metrics-certs\") pod \"maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n\" (UID: \"571da8ac-1d91-472b-80a1-187c16a687e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"
Oct 30 01:02:15.064385 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:15.064331    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw4tm\" (UniqueName: \"kubernetes.io/projected/571da8ac-1d91-472b-80a1-187c16a687e8-kube-api-access-kw4tm\") pod \"maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n\" (UID: \"571da8ac-1d91-472b-80a1-187c16a687e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"
Oct 30 01:02:15.162259 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:15.162164    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"
Oct 30 01:02:15.475883 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:02:15.475852    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571da8ac_1d91_472b_80a1_187c16a687e8.slice/crio-bdf62cfd52e22e01dceaedb2dbaffe79b367a3a1e977c9b40bfef182cf748111 WatchSource:0}: Error finding container bdf62cfd52e22e01dceaedb2dbaffe79b367a3a1e977c9b40bfef182cf748111: Status 404 returned error can't find the container with id bdf62cfd52e22e01dceaedb2dbaffe79b367a3a1e977c9b40bfef182cf748111
Oct 30 01:02:15.476311 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:15.476255    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"]
Oct 30 01:02:15.570030 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:15.569997    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:02:15.570500 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:15.570478    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:02:15.571186 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:15.571168    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:02:15.705932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:15.705898    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:02:15.707002 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:15.706978    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:02:15.708050 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:15.708025    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:02:16.606151 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:16.606112    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n" event={"ID":"571da8ac-1d91-472b-80a1-187c16a687e8","Type":"ContainerStarted","Data":"05ab48848676da1718c31a1e7d209cb25f8bcdcf8e6c1854e66fea6565d444ae"}
Oct 30 01:02:16.606374 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:16.606181    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"
Oct 30 01:02:16.606374 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:16.606208    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n" event={"ID":"571da8ac-1d91-472b-80a1-187c16a687e8","Type":"ContainerStarted","Data":"bdf62cfd52e22e01dceaedb2dbaffe79b367a3a1e977c9b40bfef182cf748111"}
Oct 30 01:02:16.651028 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:02:16.650983    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:02:16.653544 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:16.653495    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n" podStartSLOduration=2.334811209 podStartE2EDuration="2.653481611s" totalImagesPullingTime="318.670402ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:16.647901476 +0000 UTC m=+1630.504809232" watchObservedRunningTime="2026-10-30 01:02:16.653481611 +0000 UTC m=+1630.510389363"
Oct 30 01:02:22.620684 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:22.620608    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"
Oct 30 01:02:28.029645 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:28.029601    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq"]
Oct 30 01:02:28.041563 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:28.041532    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2ef568-k7dwq"]
Oct 30 01:02:28.265023 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:28.264990    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8653d44e-bb96-412b-9d37-65e7f4a64064" path="/var/lib/kubelet/pods/8653d44e-bb96-412b-9d37-65e7f4a64064/volumes"
Oct 30 01:02:34.566787 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:34.566725    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:02:42.147132 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.147085    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"]
Oct 30 01:02:42.152161 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.152130    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.200939 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.200900    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ae06fe-openshift-default-dockercfg-ccc5k\""
Oct 30 01:02:42.201548 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.201523    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"]
Oct 30 01:02:42.268070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.268027    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67037690-1455-44ab-aef7-8adb912e3077-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.268373 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.268351    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.268596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.268572    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.268755 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.268737    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.269149 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.269130    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.269326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.269299    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxcxf\" (UniqueName: \"kubernetes.io/projected/67037690-1455-44ab-aef7-8adb912e3077-kube-api-access-mxcxf\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.269472 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.269456    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67037690-1455-44ab-aef7-8adb912e3077-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.269647 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.269625    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.269821 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.269797    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67037690-1455-44ab-aef7-8adb912e3077-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.391784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.391737    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.391961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.391804    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxcxf\" (UniqueName: \"kubernetes.io/projected/67037690-1455-44ab-aef7-8adb912e3077-kube-api-access-mxcxf\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.391961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.391834    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67037690-1455-44ab-aef7-8adb912e3077-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.391961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.391887    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.391961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.391928    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67037690-1455-44ab-aef7-8adb912e3077-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.392115 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.392103    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67037690-1455-44ab-aef7-8adb912e3077-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.392172 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.392138    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.392172 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.392160    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.392257 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.392227    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.392293 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.392274    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.392618 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.392597    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.392942 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.392912    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67037690-1455-44ab-aef7-8adb912e3077-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.393114 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.393086    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.393295 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.393270    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.395385 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.395359    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67037690-1455-44ab-aef7-8adb912e3077-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.395822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.395801    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.415607 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.415535    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67037690-1455-44ab-aef7-8adb912e3077-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.415775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.415643    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxcxf\" (UniqueName: \"kubernetes.io/projected/67037690-1455-44ab-aef7-8adb912e3077-kube-api-access-mxcxf\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.506569 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.506501    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:42.928232 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:02:42.925810    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67037690_1455_44ab_aef7_8adb912e3077.slice/crio-84368adec3e825824aaec4f9900e56db14c58974f494c8f82e97348d72133c42 WatchSource:0}: Error finding container 84368adec3e825824aaec4f9900e56db14c58974f494c8f82e97348d72133c42: Status 404 returned error can't find the container with id 84368adec3e825824aaec4f9900e56db14c58974f494c8f82e97348d72133c42
Oct 30 01:02:42.928232 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.926315    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"]
Oct 30 01:02:42.929070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.929028    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:02:42.929201 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.929100    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:02:42.929201 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:42.929136    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:02:43.944424 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:43.944382    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4" event={"ID":"67037690-1455-44ab-aef7-8adb912e3077","Type":"ContainerStarted","Data":"c5d4c46681067afaec4d5a7b7b938d43b50bba061dd223d8f1259b7e13c9e23f"}
Oct 30 01:02:43.944910 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:43.944447    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4" event={"ID":"67037690-1455-44ab-aef7-8adb912e3077","Type":"ContainerStarted","Data":"84368adec3e825824aaec4f9900e56db14c58974f494c8f82e97348d72133c42"}
Oct 30 01:02:44.000930 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:44.000869    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4" podStartSLOduration=2.000852728 podStartE2EDuration="2.000852728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:43.999529284 +0000 UTC m=+1657.856437042" watchObservedRunningTime="2026-10-30 01:02:44.000852728 +0000 UTC m=+1657.857760494"
Oct 30 01:02:44.509727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:44.508260    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:44.515788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:44.515743    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:44.962295 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:44.962265    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:44.963404 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:44.963382    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:02:46.474881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:46.474845    6099 patch_prober.go:29] interesting pod/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:02:46.475282 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:46.474913    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c" podUID="502b2376-6a3e-4c6d-9cab-aa704bcd3053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:46.638632 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:02:46.638595    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:02:46.981290 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:46.981201    6099 generic.go:356] "Generic (PLEG): container finished" podID="502b2376-6a3e-4c6d-9cab-aa704bcd3053" containerID="846f119fc4eeffdcc2136c24c8139230a91c411b2c2fdad8a5891e0877a8fabb" exitCode=0
Oct 30 01:02:46.981290 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:46.981268    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c" event={"ID":"502b2376-6a3e-4c6d-9cab-aa704bcd3053","Type":"ContainerDied","Data":"846f119fc4eeffdcc2136c24c8139230a91c411b2c2fdad8a5891e0877a8fabb"}
Oct 30 01:02:46.981877 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:46.981862    6099 scope.go:122] "RemoveContainer" containerID="846f119fc4eeffdcc2136c24c8139230a91c411b2c2fdad8a5891e0877a8fabb"
Oct 30 01:02:46.983788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:46.983760    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:02:46.983860 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:46.983830    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:02:46.983903 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:46.983868    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:02:48.016952 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:48.016909    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c" event={"ID":"502b2376-6a3e-4c6d-9cab-aa704bcd3053","Type":"ContainerStarted","Data":"6a4b44f6633b75b933f8c95ab507ac883796deab0fe5b7cc83ac1985234f43d3"}
Oct 30 01:02:48.296288 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:48.296252    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4"]
Oct 30 01:02:48.307510 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:48.306332    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4"
Oct 30 01:02:48.309996 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:48.309962    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4"]
Oct 30 01:02:48.320954 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:48.320910    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-ae06fe-dockercfg-dl2rs\""
Oct 30 01:02:48.321143 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:48.321038    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ae06fe\""
Oct 30 01:02:48.334688 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:48.333270    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:02:48.334688 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:48.333801    6099 patch_prober.go:29] interesting pod/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 30 01:02:48.334688 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:48.333890    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c" podUID="502b2376-6a3e-4c6d-9cab-aa704bcd3053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 30 01:02:48.337052 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:48.337024    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293c28e2-6799-412b-9373-1e418269cd10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4\" (UID: \"293c28e2-6799-412b-9373-1e418269cd10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4"
Oct 30 01:02:48.337306 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:48.337282    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ng8r\" (UniqueName: \"kubernetes.io/projected/293c28e2-6799-412b-9373-1e418269cd10-kube-api-access-9ng8r\") pod \"payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4\" (UID: \"293c28e2-6799-412b-9373-1e418269cd10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4"
Oct 30 01:02:48.441757 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:48.439308    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ng8r\" (UniqueName: \"kubernetes.io/projected/293c28e2-6799-412b-9373-1e418269cd10-kube-api-access-9ng8r\") pod \"payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4\" (UID: \"293c28e2-6799-412b-9373-1e418269cd10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4"
Oct 30 01:02:48.441757 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:48.439609    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293c28e2-6799-412b-9373-1e418269cd10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4\" (UID: \"293c28e2-6799-412b-9373-1e418269cd10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4"
Oct 30 01:02:48.441757 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:48.440833    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293c28e2-6799-412b-9373-1e418269cd10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4\" (UID: \"293c28e2-6799-412b-9373-1e418269cd10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4"
Oct 30 01:02:48.488729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:48.481557    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ng8r\" (UniqueName: \"kubernetes.io/projected/293c28e2-6799-412b-9373-1e418269cd10-kube-api-access-9ng8r\") pod \"payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4\" (UID: \"293c28e2-6799-412b-9373-1e418269cd10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4"
Oct 30 01:02:48.626526 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:48.626439    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4"
Oct 30 01:02:49.132723 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:49.132671    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4"]
Oct 30 01:02:49.134549 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:02:49.133660    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293c28e2_6799_412b_9373_1e418269cd10.slice/crio-4ac632259085644ebe659bc5fc91183194f3066279bb368d6b506f8f18b2a35c WatchSource:0}: Error finding container 4ac632259085644ebe659bc5fc91183194f3066279bb368d6b506f8f18b2a35c: Status 404 returned error can't find the container with id 4ac632259085644ebe659bc5fc91183194f3066279bb368d6b506f8f18b2a35c
Oct 30 01:02:49.333442 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:49.333402    6099 patch_prober.go:29] interesting pod/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 30 01:02:49.333592 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:49.333464    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c" podUID="502b2376-6a3e-4c6d-9cab-aa704bcd3053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 30 01:02:50.037046 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:50.037003    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4" event={"ID":"293c28e2-6799-412b-9373-1e418269cd10","Type":"ContainerStarted","Data":"813f5334bfbabc156b72150f61202f621b6a85a7b2dedab9119f67eec7415de8"}
Oct 30 01:02:50.037238 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:50.037093    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4" event={"ID":"293c28e2-6799-412b-9373-1e418269cd10","Type":"ContainerStarted","Data":"4ac632259085644ebe659bc5fc91183194f3066279bb368d6b506f8f18b2a35c"}
Oct 30 01:02:50.037238 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:50.037120    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4"
Oct 30 01:02:50.091285 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:50.091218    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4" podStartSLOduration=2.09119971 podStartE2EDuration="2.09119971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:50.09037562 +0000 UTC m=+1663.947283376" watchObservedRunningTime="2026-10-30 01:02:50.09119971 +0000 UTC m=+1663.948107467"
Oct 30 01:02:50.333412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:50.333371    6099 patch_prober.go:29] interesting pod/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 30 01:02:50.333916 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:50.333446    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c" podUID="502b2376-6a3e-4c6d-9cab-aa704bcd3053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 30 01:02:51.333509 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:51.333470    6099 patch_prober.go:29] interesting pod/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 30 01:02:51.334055 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:51.333554    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c" podUID="502b2376-6a3e-4c6d-9cab-aa704bcd3053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 30 01:02:52.333319 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:52.333287    6099 patch_prober.go:29] interesting pod/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 30 01:02:52.333507 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:52.333351    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c" podUID="502b2376-6a3e-4c6d-9cab-aa704bcd3053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 30 01:02:53.333309 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:53.333273    6099 patch_prober.go:29] interesting pod/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 30 01:02:53.333755 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:53.333337    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c" podUID="502b2376-6a3e-4c6d-9cab-aa704bcd3053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 30 01:02:54.337898 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:54.337865    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:02:54.338387 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:54.338365    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:02:54.339123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:54.339101    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:02:56.857794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:56.857761    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"]
Oct 30 01:02:56.861636 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:56.861611    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:56.887567 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:56.887539    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"]
Oct 30 01:02:56.904024 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:56.903997    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-97ceef-openshift-default-dockercfg-zqr8k\""
Oct 30 01:02:57.029790 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.017110    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb207b92-e133-428b-99b2-e17d2e903511-istiod-ca-cert\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.029790 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.017179    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-workload-certs\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.029790 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.017279    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-workload-socket\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.029790 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.017304    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb207b92-e133-428b-99b2-e17d2e903511-istio-podinfo\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.029790 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.017358    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-istio-envoy\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.029790 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.017419    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb207b92-e133-428b-99b2-e17d2e903511-istio-token\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.029790 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.017491    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-istio-data\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.029790 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.017576    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-credential-socket\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.029790 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.017622    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xvtd\" (UniqueName: \"kubernetes.io/projected/cb207b92-e133-428b-99b2-e17d2e903511-kube-api-access-9xvtd\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.131370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.131278    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb207b92-e133-428b-99b2-e17d2e903511-istiod-ca-cert\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.131370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.131355    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-workload-certs\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.131557 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.131456    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-workload-socket\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.131557 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.131480    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb207b92-e133-428b-99b2-e17d2e903511-istio-podinfo\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.131557 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.131534    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-istio-envoy\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.131658 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.131601    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb207b92-e133-428b-99b2-e17d2e903511-istio-token\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.131696 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.131671    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-istio-data\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.132357 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.132328    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-credential-socket\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.132426 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.132406    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xvtd\" (UniqueName: \"kubernetes.io/projected/cb207b92-e133-428b-99b2-e17d2e903511-kube-api-access-9xvtd\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.133587 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.133555    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-workload-certs\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.133732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.133596    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-istio-data\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.133978 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.133951    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-credential-socket\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.134072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.134035    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-workload-socket\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.134296 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.134272    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb207b92-e133-428b-99b2-e17d2e903511-istiod-ca-cert\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.136530 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.136504    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb207b92-e133-428b-99b2-e17d2e903511-istio-podinfo\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.144266 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.144242    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-istio-envoy\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.158618 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.158588    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb207b92-e133-428b-99b2-e17d2e903511-istio-token\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.159460 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.159433    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xvtd\" (UniqueName: \"kubernetes.io/projected/cb207b92-e133-428b-99b2-e17d2e903511-kube-api-access-9xvtd\") pod \"e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") " pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.182779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.182746    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:57.534786 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:02:57.534752    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb207b92_e133_428b_99b2_e17d2e903511.slice/crio-cccb2d83678a473231acc92c027201b19cebb3d57d20e5a58c3dca27e26cca2f WatchSource:0}: Error finding container cccb2d83678a473231acc92c027201b19cebb3d57d20e5a58c3dca27e26cca2f: Status 404 returned error can't find the container with id cccb2d83678a473231acc92c027201b19cebb3d57d20e5a58c3dca27e26cca2f
Oct 30 01:02:57.535328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.535294    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"]
Oct 30 01:02:57.537966 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.537929    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:02:57.538098 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.538014    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:02:57.538098 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:57.538060    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:02:58.144066 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.144022    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5" event={"ID":"cb207b92-e133-428b-99b2-e17d2e903511","Type":"ContainerStarted","Data":"3f9aa1642154abcfdac8ab90306762bc06602008b969c02c9c6bb449a6320bb7"}
Oct 30 01:02:58.144475 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.144085    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5" event={"ID":"cb207b92-e133-428b-99b2-e17d2e903511","Type":"ContainerStarted","Data":"cccb2d83678a473231acc92c027201b19cebb3d57d20e5a58c3dca27e26cca2f"}
Oct 30 01:02:58.183168 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.183137    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:58.189586 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.189559    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:58.206812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.206744    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5" podStartSLOduration=2.206726476 podStartE2EDuration="2.206726476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:58.199697064 +0000 UTC m=+1672.056604813" watchObservedRunningTime="2026-10-30 01:02:58.206726476 +0000 UTC m=+1672.063634232"
Oct 30 01:02:58.596138 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.596105    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"]
Oct 30 01:02:58.600098 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.600076    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"
Oct 30 01:02:58.623637 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.623608    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"]
Oct 30 01:02:58.625563 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.625537    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-dd257f-serving-cert\""
Oct 30 01:02:58.760996 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.760961    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70cef62b-b6ff-4aca-9ba3-680d50316121-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7\" (UID: \"70cef62b-b6ff-4aca-9ba3-680d50316121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"
Oct 30 01:02:58.761196 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.761165    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvq5x\" (UniqueName: \"kubernetes.io/projected/70cef62b-b6ff-4aca-9ba3-680d50316121-kube-api-access-lvq5x\") pod \"maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7\" (UID: \"70cef62b-b6ff-4aca-9ba3-680d50316121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"
Oct 30 01:02:58.761347 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.761323    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70cef62b-b6ff-4aca-9ba3-680d50316121-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7\" (UID: \"70cef62b-b6ff-4aca-9ba3-680d50316121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"
Oct 30 01:02:58.761397 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.761370    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70cef62b-b6ff-4aca-9ba3-680d50316121-metrics-certs\") pod \"maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7\" (UID: \"70cef62b-b6ff-4aca-9ba3-680d50316121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"
Oct 30 01:02:58.873779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.866606    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvq5x\" (UniqueName: \"kubernetes.io/projected/70cef62b-b6ff-4aca-9ba3-680d50316121-kube-api-access-lvq5x\") pod \"maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7\" (UID: \"70cef62b-b6ff-4aca-9ba3-680d50316121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"
Oct 30 01:02:58.873779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.866841    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70cef62b-b6ff-4aca-9ba3-680d50316121-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7\" (UID: \"70cef62b-b6ff-4aca-9ba3-680d50316121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"
Oct 30 01:02:58.873779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.866894    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70cef62b-b6ff-4aca-9ba3-680d50316121-metrics-certs\") pod \"maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7\" (UID: \"70cef62b-b6ff-4aca-9ba3-680d50316121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"
Oct 30 01:02:58.873779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.871032    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70cef62b-b6ff-4aca-9ba3-680d50316121-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7\" (UID: \"70cef62b-b6ff-4aca-9ba3-680d50316121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"
Oct 30 01:02:58.873779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.869547    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70cef62b-b6ff-4aca-9ba3-680d50316121-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7\" (UID: \"70cef62b-b6ff-4aca-9ba3-680d50316121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"
Oct 30 01:02:58.886458 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.886419    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70cef62b-b6ff-4aca-9ba3-680d50316121-metrics-certs\") pod \"maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7\" (UID: \"70cef62b-b6ff-4aca-9ba3-680d50316121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"
Oct 30 01:02:58.892610 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.892575    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70cef62b-b6ff-4aca-9ba3-680d50316121-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7\" (UID: \"70cef62b-b6ff-4aca-9ba3-680d50316121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"
Oct 30 01:02:58.897274 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.897241    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvq5x\" (UniqueName: \"kubernetes.io/projected/70cef62b-b6ff-4aca-9ba3-680d50316121-kube-api-access-lvq5x\") pod \"maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7\" (UID: \"70cef62b-b6ff-4aca-9ba3-680d50316121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"
Oct 30 01:02:58.915090 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:58.915059    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"
Oct 30 01:02:59.154272 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:59.154191    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:59.155166 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:59.155145    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:02:59.254397 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:02:59.254351    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"]
Oct 30 01:02:59.259893 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:02:59.259861    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70cef62b_b6ff_4aca_9ba3_680d50316121.slice/crio-f99ebb064fc41a3cf19746e85136ef81ae96d8d2cc69cd7c9d371aaca802e42f WatchSource:0}: Error finding container f99ebb064fc41a3cf19746e85136ef81ae96d8d2cc69cd7c9d371aaca802e42f: Status 404 returned error can't find the container with id f99ebb064fc41a3cf19746e85136ef81ae96d8d2cc69cd7c9d371aaca802e42f
Oct 30 01:03:00.208841 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:00.208803    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7" event={"ID":"70cef62b-b6ff-4aca-9ba3-680d50316121","Type":"ContainerStarted","Data":"3f4a824958a1aba34f129c415f620341c7b5f216257dfee78fca068bd919722d"}
Oct 30 01:03:00.209351 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:00.208867    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7" event={"ID":"70cef62b-b6ff-4aca-9ba3-680d50316121","Type":"ContainerStarted","Data":"f99ebb064fc41a3cf19746e85136ef81ae96d8d2cc69cd7c9d371aaca802e42f"}
Oct 30 01:03:00.209351 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:00.209045    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"
Oct 30 01:03:00.251544 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:00.251491    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7" podStartSLOduration=1.9453526380000001 podStartE2EDuration="2.251478572s" totalImagesPullingTime="306.125934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:00.24963087 +0000 UTC m=+1674.106538628" watchObservedRunningTime="2026-10-30 01:03:00.251478572 +0000 UTC m=+1674.108386327"
Oct 30 01:03:01.050060 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:01.050030    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4"
Oct 30 01:03:03.181366 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:03.181336    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj"]
Oct 30 01:03:03.185927 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:03.185903    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj"
Oct 30 01:03:03.207057 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:03.206988    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-97ceef\""
Oct 30 01:03:03.211940 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:03.211916    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj"]
Oct 30 01:03:03.248284 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:03.248249    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55btb\" (UniqueName: \"kubernetes.io/projected/497a9701-e362-48d7-a2cd-cfa2a5a1e535-kube-api-access-55btb\") pod \"payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj\" (UID: \"497a9701-e362-48d7-a2cd-cfa2a5a1e535\") " pod="openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj"
Oct 30 01:03:03.248489 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:03.248471    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497a9701-e362-48d7-a2cd-cfa2a5a1e535-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj\" (UID: \"497a9701-e362-48d7-a2cd-cfa2a5a1e535\") " pod="openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj"
Oct 30 01:03:03.382771 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:03.355988    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497a9701-e362-48d7-a2cd-cfa2a5a1e535-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj\" (UID: \"497a9701-e362-48d7-a2cd-cfa2a5a1e535\") " pod="openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj"
Oct 30 01:03:03.382771 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:03.379203    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55btb\" (UniqueName: \"kubernetes.io/projected/497a9701-e362-48d7-a2cd-cfa2a5a1e535-kube-api-access-55btb\") pod \"payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj\" (UID: \"497a9701-e362-48d7-a2cd-cfa2a5a1e535\") " pod="openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj"
Oct 30 01:03:03.386659 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:03.386623    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497a9701-e362-48d7-a2cd-cfa2a5a1e535-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj\" (UID: \"497a9701-e362-48d7-a2cd-cfa2a5a1e535\") " pod="openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj"
Oct 30 01:03:03.459530 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:03.459452    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55btb\" (UniqueName: \"kubernetes.io/projected/497a9701-e362-48d7-a2cd-cfa2a5a1e535-kube-api-access-55btb\") pod \"payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj\" (UID: \"497a9701-e362-48d7-a2cd-cfa2a5a1e535\") " pod="openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj"
Oct 30 01:03:03.504044 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:03.504006    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj"
Oct 30 01:03:03.852757 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:03.852012    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj"]
Oct 30 01:03:03.853175 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:03:03.853148    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497a9701_e362_48d7_a2cd_cfa2a5a1e535.slice/crio-f7e3e24fdd1948c747f4c68986107ea976efd1cc3e6df084ac2594d4ad8dfe22 WatchSource:0}: Error finding container f7e3e24fdd1948c747f4c68986107ea976efd1cc3e6df084ac2594d4ad8dfe22: Status 404 returned error can't find the container with id f7e3e24fdd1948c747f4c68986107ea976efd1cc3e6df084ac2594d4ad8dfe22
Oct 30 01:03:04.251907 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:04.251815    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj" event={"ID":"497a9701-e362-48d7-a2cd-cfa2a5a1e535","Type":"ContainerStarted","Data":"d1962a41ec1d97640e65171378146e0849ce9d3de0a7674bcd4c08044fce7624"}
Oct 30 01:03:04.251907 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:04.251874    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj" event={"ID":"497a9701-e362-48d7-a2cd-cfa2a5a1e535","Type":"ContainerStarted","Data":"f7e3e24fdd1948c747f4c68986107ea976efd1cc3e6df084ac2594d4ad8dfe22"}
Oct 30 01:03:04.252336 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:04.252024    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj"
Oct 30 01:03:04.296245 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:04.296190    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj" podStartSLOduration=1.296176995 podStartE2EDuration="1.296176995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:04.292597338 +0000 UTC m=+1678.149505093" watchObservedRunningTime="2026-10-30 01:03:04.296176995 +0000 UTC m=+1678.153084750"
Oct 30 01:03:06.229167 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:06.229135    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"
Oct 30 01:03:11.755932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:11.755901    6099 scope.go:122] "RemoveContainer" containerID="99be46bd352a142f44fe3095772b251e059bfd23a894060cbe940931c8818d13"
Oct 30 01:03:11.768046 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:11.768027    6099 scope.go:122] "RemoveContainer" containerID="43864f08ada5a10e5a6374101fd98d1f6203ed4faa2044b2062ed0f5214433d5"
Oct 30 01:03:15.268859 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:15.268828    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj"
Oct 30 01:03:16.671133 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:03:16.671096    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:03:21.911727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:21.910758    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:03:43.305745 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:43.305678    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal" podUID="f8c3a9edbd8cbf03175beab8c4db7242" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 01:03:45.026926 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.026886    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"]
Oct 30 01:03:45.031537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.031517    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.046022 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.046000    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-43c02d-openshift-default-dockercfg-q4fqr\""
Oct 30 01:03:45.063889 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.063856    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"]
Oct 30 01:03:45.146191 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.146161    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbad3465-dc78-4586-a4dc-071709ad50dd-istiod-ca-cert\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.146191 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.146197    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-credential-socket\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.146420 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.146317    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-data\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.146420 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.146401    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-token\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.146505 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.146468    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-workload-certs\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.146555 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.146527    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-workload-socket\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.146671 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.146652    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-envoy\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.146806 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.146791    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdtf2\" (UniqueName: \"kubernetes.io/projected/fbad3465-dc78-4586-a4dc-071709ad50dd-kube-api-access-rdtf2\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.146854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.146834    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-podinfo\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.248352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.248305    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-workload-socket\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.248564 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.248452    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-envoy\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.248634 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.248588    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdtf2\" (UniqueName: \"kubernetes.io/projected/fbad3465-dc78-4586-a4dc-071709ad50dd-kube-api-access-rdtf2\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.248724 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.248645    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-podinfo\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.248724 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.248685    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbad3465-dc78-4586-a4dc-071709ad50dd-istiod-ca-cert\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.248859 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.248765    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-credential-socket\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.248859 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.248851    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-data\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.249050 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.249016    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-token\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.249194 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.249109    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-workload-certs\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.249436 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.248764    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-workload-socket\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.249673 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.249481    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-workload-certs\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.249673 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.249517    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbad3465-dc78-4586-a4dc-071709ad50dd-istiod-ca-cert\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.249892 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.249682    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-data\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.250230 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.250200    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-credential-socket\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.251392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.251365    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-envoy\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.251574 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.251555    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-podinfo\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.272107 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.272078    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-token\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.272247 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.272227    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdtf2\" (UniqueName: \"kubernetes.io/projected/fbad3465-dc78-4586-a4dc-071709ad50dd-kube-api-access-rdtf2\") pod \"e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") " pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.346595 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.346560    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:45.665763 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:03:45.665731    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbad3465_dc78_4586_a4dc_071709ad50dd.slice/crio-77a4872cb382ccd4e8a37985bdc2018cbb7ee296dc7807f0ba4297b962e15bbe WatchSource:0}: Error finding container 77a4872cb382ccd4e8a37985bdc2018cbb7ee296dc7807f0ba4297b962e15bbe: Status 404 returned error can't find the container with id 77a4872cb382ccd4e8a37985bdc2018cbb7ee296dc7807f0ba4297b962e15bbe
Oct 30 01:03:45.666333 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.666309    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"]
Oct 30 01:03:45.668569 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.668534    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:03:45.668683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.668614    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:03:45.668683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.668661    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:03:45.829451 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.829366    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t" event={"ID":"fbad3465-dc78-4586-a4dc-071709ad50dd","Type":"ContainerStarted","Data":"e52e84720efbc78c384abcb7bdcde2ff362e292f8080e637fdab1d64289a55c5"}
Oct 30 01:03:45.829451 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.829433    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t" event={"ID":"fbad3465-dc78-4586-a4dc-071709ad50dd","Type":"ContainerStarted","Data":"77a4872cb382ccd4e8a37985bdc2018cbb7ee296dc7807f0ba4297b962e15bbe"}
Oct 30 01:03:45.874088 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:45.874037    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t" podStartSLOduration=1.874024011 podStartE2EDuration="1.874024011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:45.872268538 +0000 UTC m=+1719.729176293" watchObservedRunningTime="2026-10-30 01:03:45.874024011 +0000 UTC m=+1719.730931765"
Oct 30 01:03:46.346744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:46.346677    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:46.348256 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:46.348230    6099 patch_prober.go:29] interesting pod/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 01:03:46.348381 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:46.348276    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t" podUID="fbad3465-dc78-4586-a4dc-071709ad50dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 01:03:46.634312 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:03:46.634227    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:03:47.346900 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:47.346864    6099 patch_prober.go:29] interesting pod/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 01:03:47.347389 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:47.346917    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t" podUID="fbad3465-dc78-4586-a4dc-071709ad50dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 01:03:48.347185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.347093    6099 patch_prober.go:29] interesting pod/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 01:03:48.347185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.347153    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t" podUID="fbad3465-dc78-4586-a4dc-071709ad50dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 01:03:48.421931 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.421897    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"]
Oct 30 01:03:48.425970 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.425948    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.455584 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.455551    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"]
Oct 30 01:03:48.493338 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.493306    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5f4f9b-openshift-default-dockercfg-w9b79\""
Oct 30 01:03:48.525463 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.525427    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqblp\" (UniqueName: \"kubernetes.io/projected/394e8301-d15d-46d4-aa5c-10f3e154c948-kube-api-access-kqblp\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.525670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.525628    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394e8301-d15d-46d4-aa5c-10f3e154c948-istiod-ca-cert\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.525670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.525665    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-workload-certs\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.525869 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.525814    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-podinfo\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.525935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.525877    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-token\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.526011 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.525994    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-data\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.526072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.526042    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-workload-socket\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.526142 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.526125    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-envoy\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.526206 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.526175    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-credential-socket\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.627519 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.627425    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-data\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.627519 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.627482    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-workload-socket\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.627760 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.627527    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-envoy\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.627760 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.627565    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-credential-socket\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.627760 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.627730    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqblp\" (UniqueName: \"kubernetes.io/projected/394e8301-d15d-46d4-aa5c-10f3e154c948-kube-api-access-kqblp\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.627938 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.627774    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-data\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.627938 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.627892    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-workload-socket\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.627938 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.627896    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394e8301-d15d-46d4-aa5c-10f3e154c948-istiod-ca-cert\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.628119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.627971    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-workload-certs\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.628119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.628071    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-podinfo\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.628119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.627973    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-credential-socket\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.628119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.628106    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-token\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.628333 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.628232    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-workload-certs\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.628551 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.628529    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394e8301-d15d-46d4-aa5c-10f3e154c948-istiod-ca-cert\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.629932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.629901    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-envoy\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.630318 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.630297    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-podinfo\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.658319 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.658293    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-token\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.658586 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.658566    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqblp\" (UniqueName: \"kubernetes.io/projected/394e8301-d15d-46d4-aa5c-10f3e154c948-kube-api-access-kqblp\") pod \"e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") " pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:48.742213 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:48.742174    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:49.102491 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:03:49.102458    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394e8301_d15d_46d4_aa5c_10f3e154c948.slice/crio-3b24782bdd16ec5dc265f30ba0ba8ae4347c5ab4a5160b99ed1d5c655ef4f9f2 WatchSource:0}: Error finding container 3b24782bdd16ec5dc265f30ba0ba8ae4347c5ab4a5160b99ed1d5c655ef4f9f2: Status 404 returned error can't find the container with id 3b24782bdd16ec5dc265f30ba0ba8ae4347c5ab4a5160b99ed1d5c655ef4f9f2
Oct 30 01:03:49.102932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:49.102909    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"]
Oct 30 01:03:49.105617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:49.105582    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:03:49.105765 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:49.105650    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:03:49.105765 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:49.105681    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:03:49.347071 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:49.347038    6099 patch_prober.go:29] interesting pod/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 01:03:49.347254 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:49.347091    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t" podUID="fbad3465-dc78-4586-a4dc-071709ad50dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 01:03:49.918754 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:49.918041    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226" event={"ID":"394e8301-d15d-46d4-aa5c-10f3e154c948","Type":"ContainerStarted","Data":"2ab1008466a25a41384bc38412922c0c0fb9dc7bdb920a79f11f01d287ada0bb"}
Oct 30 01:03:49.918754 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:49.918094    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226" event={"ID":"394e8301-d15d-46d4-aa5c-10f3e154c948","Type":"ContainerStarted","Data":"3b24782bdd16ec5dc265f30ba0ba8ae4347c5ab4a5160b99ed1d5c655ef4f9f2"}
Oct 30 01:03:50.025737 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:50.025352    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226" podStartSLOduration=2.025329976 podStartE2EDuration="2.025329976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:49.986091002 +0000 UTC m=+1723.842998756" watchObservedRunningTime="2026-10-30 01:03:50.025329976 +0000 UTC m=+1723.882237734"
Oct 30 01:03:50.366762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:50.363819    6099 patch_prober.go:29] interesting pod/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 01:03:50.366762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:50.363879    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t" podUID="fbad3465-dc78-4586-a4dc-071709ad50dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 01:03:50.753569 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:50.753474    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:50.756865 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:50.756832    6099 patch_prober.go:29] interesting pod/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 01:03:50.759614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:50.759579    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226" podUID="394e8301-d15d-46d4-aa5c-10f3e154c948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 01:03:51.346867 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:51.346830    6099 patch_prober.go:29] interesting pod/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 01:03:51.347054 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:51.346883    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t" podUID="fbad3465-dc78-4586-a4dc-071709ad50dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 01:03:51.742885 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:51.742786    6099 patch_prober.go:29] interesting pod/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 01:03:51.742885 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:51.742861    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226" podUID="394e8301-d15d-46d4-aa5c-10f3e154c948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 01:03:52.346970 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:52.346937    6099 patch_prober.go:29] interesting pod/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 01:03:52.347159 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:52.347002    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t" podUID="fbad3465-dc78-4586-a4dc-071709ad50dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 01:03:52.743620 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:52.743473    6099 patch_prober.go:29] interesting pod/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 01:03:52.744068 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:52.743557    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226" podUID="394e8301-d15d-46d4-aa5c-10f3e154c948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 01:03:53.353347 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:53.353319    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:53.353832 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:53.353806    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:53.354597 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:53.354579    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:03:53.743474 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:53.743389    6099 patch_prober.go:29] interesting pod/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 01:03:53.743474 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:53.743441    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226" podUID="394e8301-d15d-46d4-aa5c-10f3e154c948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 01:03:54.036809 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:54.036729    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8"]
Oct 30 01:03:54.047732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:54.047694    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-416f373e-vxlm8"]
Oct 30 01:03:54.289781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:54.264867    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7584e7cf-bf67-4082-adb9-415425b02f9d" path="/var/lib/kubelet/pods/7584e7cf-bf67-4082-adb9-415425b02f9d/volumes"
Oct 30 01:03:54.743249 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:54.743165    6099 patch_prober.go:29] interesting pod/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 01:03:54.743249 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:54.743220    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226" podUID="394e8301-d15d-46d4-aa5c-10f3e154c948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 01:03:55.742790 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:55.742700    6099 patch_prober.go:29] interesting pod/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 01:03:55.743198 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:55.742817    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226" podUID="394e8301-d15d-46d4-aa5c-10f3e154c948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 01:03:56.748943 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:56.748911    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:56.749422 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:56.749402    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:03:56.750085 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:03:56.750059    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:04:05.062312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:05.062277    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"]
Oct 30 01:04:05.071245 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:05.071174    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:04:05.096400 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:05.096363    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"]
Oct 30 01:04:05.098980 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:05.098952    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f96ede-serving-cert\""
Oct 30 01:04:05.155922 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:05.155878    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7187ea-003c-49c3-a37f-1f4103649d1d-metrics-certs\") pod \"maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb\" (UID: \"7c7187ea-003c-49c3-a37f-1f4103649d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:04:05.156342 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:05.156313    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7187ea-003c-49c3-a37f-1f4103649d1d-maas-api-tls\") pod \"maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb\" (UID: \"7c7187ea-003c-49c3-a37f-1f4103649d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:04:05.156478 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:05.156355    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7187ea-003c-49c3-a37f-1f4103649d1d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb\" (UID: \"7c7187ea-003c-49c3-a37f-1f4103649d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:04:05.156672 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:05.156645    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frwp2\" (UniqueName: \"kubernetes.io/projected/7c7187ea-003c-49c3-a37f-1f4103649d1d-kube-api-access-frwp2\") pod \"maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb\" (UID: \"7c7187ea-003c-49c3-a37f-1f4103649d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:04:05.258559 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:05.258515    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7187ea-003c-49c3-a37f-1f4103649d1d-maas-api-tls\") pod \"maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb\" (UID: \"7c7187ea-003c-49c3-a37f-1f4103649d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:04:05.258808 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:05.258572    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7187ea-003c-49c3-a37f-1f4103649d1d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb\" (UID: \"7c7187ea-003c-49c3-a37f-1f4103649d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:04:05.259952 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:04:05.259198    6099 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-serving-cert: secret "maas-api-e2e-inf-b-f96ede-serving-cert" not found
Oct 30 01:04:05.259952 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:04:05.259301    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c7187ea-003c-49c3-a37f-1f4103649d1d-maas-api-tls podName:7c7187ea-003c-49c3-a37f-1f4103649d1d nodeName:}" failed. No retries permitted until 2026-10-30 01:04:05.759272965 +0000 UTC m=+1739.616180700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c7187ea-003c-49c3-a37f-1f4103649d1d-maas-api-tls") pod "maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb" (UID: "7c7187ea-003c-49c3-a37f-1f4103649d1d") : secret "maas-api-e2e-inf-b-f96ede-serving-cert" not found
Oct 30 01:04:05.260160 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:05.259953    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7187ea-003c-49c3-a37f-1f4103649d1d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb\" (UID: \"7c7187ea-003c-49c3-a37f-1f4103649d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:04:05.272895 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:05.260625    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwp2\" (UniqueName: \"kubernetes.io/projected/7c7187ea-003c-49c3-a37f-1f4103649d1d-kube-api-access-frwp2\") pod \"maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb\" (UID: \"7c7187ea-003c-49c3-a37f-1f4103649d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:04:05.272895 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:05.260789    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7187ea-003c-49c3-a37f-1f4103649d1d-metrics-certs\") pod \"maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb\" (UID: \"7c7187ea-003c-49c3-a37f-1f4103649d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:04:05.287922 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:05.286773    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frwp2\" (UniqueName: \"kubernetes.io/projected/7c7187ea-003c-49c3-a37f-1f4103649d1d-kube-api-access-frwp2\") pod \"maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb\" (UID: \"7c7187ea-003c-49c3-a37f-1f4103649d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:04:05.290558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:05.290525    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7187ea-003c-49c3-a37f-1f4103649d1d-metrics-certs\") pod \"maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb\" (UID: \"7c7187ea-003c-49c3-a37f-1f4103649d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:04:05.769684 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:05.769651    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7187ea-003c-49c3-a37f-1f4103649d1d-maas-api-tls\") pod \"maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb\" (UID: \"7c7187ea-003c-49c3-a37f-1f4103649d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:04:05.772015 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:05.771994    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7187ea-003c-49c3-a37f-1f4103649d1d-maas-api-tls\") pod \"maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb\" (UID: \"7c7187ea-003c-49c3-a37f-1f4103649d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:04:05.999674 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:05.999632    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:04:06.312481 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:04:06.312415    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7187ea_003c_49c3_a37f_1f4103649d1d.slice/crio-28e2f3a614187af37749ec39532ba077f749ea6feef3e5875bac7cb723a7c9be WatchSource:0}: Error finding container 28e2f3a614187af37749ec39532ba077f749ea6feef3e5875bac7cb723a7c9be: Status 404 returned error can't find the container with id 28e2f3a614187af37749ec39532ba077f749ea6feef3e5875bac7cb723a7c9be
Oct 30 01:04:06.313274 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:06.313244    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"]
Oct 30 01:04:07.082195 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:07.082159    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb" event={"ID":"7c7187ea-003c-49c3-a37f-1f4103649d1d","Type":"ContainerStarted","Data":"92b844dff7c991e808250d08c2c1bd3dae2f250595701a0cdc6559b840a30dc2"}
Oct 30 01:04:07.082415 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:07.082216    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb" event={"ID":"7c7187ea-003c-49c3-a37f-1f4103649d1d","Type":"ContainerStarted","Data":"28e2f3a614187af37749ec39532ba077f749ea6feef3e5875bac7cb723a7c9be"}
Oct 30 01:04:07.082415 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:07.082244    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:04:07.127568 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:07.127509    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb" podStartSLOduration=1.7565746309999999 podStartE2EDuration="2.127490801s" totalImagesPullingTime="370.91617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:07.123390736 +0000 UTC m=+1740.980298492" watchObservedRunningTime="2026-10-30 01:04:07.127490801 +0000 UTC m=+1740.984398557"
Oct 30 01:04:11.872967 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:11.872935    6099 scope.go:122] "RemoveContainer" containerID="0c222070dd5cccb532cacf30b3b36ad69ff9050939c479ff8f768f23370a66cb"
Oct 30 01:04:11.884159 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:11.884134    6099 scope.go:122] "RemoveContainer" containerID="2a39584e87f13ae722a2b9cb93dfa5393d7b2511db18f420af6e769dae1119b0"
Oct 30 01:04:13.097288 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:13.097258    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:04:13.246004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:13.245884    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:04:16.645413 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:04:16.645371    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:04:27.571670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:27.571630    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"]
Oct 30 01:04:27.872794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:27.872688    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"]
Oct 30 01:04:27.872955 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:27.872806    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:27.987904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:27.987865    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slmf4\" (UniqueName: \"kubernetes.io/projected/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-kube-api-access-slmf4\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:27.988088 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:27.988070    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:27.988202 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:27.988178    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:27.988244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:27.988220    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:27.988381 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:27.988361    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:27.988432 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:27.988400    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:27.988482 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:27.988436    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:27.988557 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:27.988544    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:27.988593 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:27.988585    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.089783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.089737    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.089973 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.089835    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.089973 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.089857    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.089973 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.089951    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.090172 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.089980    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.090172 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.090015    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.090172 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.090130    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.090351 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.090194    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.090351 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.090292    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slmf4\" (UniqueName: \"kubernetes.io/projected/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-kube-api-access-slmf4\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.093728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.093595    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.093728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.093619    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.093728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.093685    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.093728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.093766    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.094101 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.094032    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.094213 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.094190    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.094381 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.094363    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.113871 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.113842    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.114173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.114147    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slmf4\" (UniqueName: \"kubernetes.io/projected/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-kube-api-access-slmf4\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:28.202164 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:28.202069    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:29.437292 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:29.437262    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"]
Oct 30 01:04:29.450861 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:29.450805    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:04:29.450991 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:29.450970    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:04:29.451219 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:29.451070    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:04:29.508270 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:29.508231    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp" event={"ID":"51be8bc2-5d53-49d4-8ae0-cd281d8d511c","Type":"ContainerStarted","Data":"1188d18bab63dc42ab694b6faf86425b2829880c3d2d0931411f25a10832c1e3"}
Oct 30 01:04:30.195890 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:30.195849    6099 patch_prober.go:29] interesting pod/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:04:30.196136 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:30.195915    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5" podUID="cb207b92-e133-428b-99b2-e17d2e903511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:30.532565 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:30.532522    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp" event={"ID":"51be8bc2-5d53-49d4-8ae0-cd281d8d511c","Type":"ContainerStarted","Data":"214e6784288ec9a99a85b4c186acad0137f01d81f5d9afadc5419f36905ea5f6"}
Oct 30 01:04:30.591202 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:30.591139    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp" podStartSLOduration=3.591121625 podStartE2EDuration="3.591121625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:30.589075834 +0000 UTC m=+1764.445983585" watchObservedRunningTime="2026-10-30 01:04:30.591121625 +0000 UTC m=+1764.448029380"
Oct 30 01:04:31.030727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:31.024457    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:04:31.030727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:31.027568    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4" podUID="67037690-1455-44ab-aef7-8adb912e3077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:04:31.203302 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:31.203259    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:31.204911 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:31.204866    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 01:04:31.205054 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:31.204927    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp" podUID="51be8bc2-5d53-49d4-8ae0-cd281d8d511c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 01:04:32.203298 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:32.203258    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 01:04:32.203935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:32.203323    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp" podUID="51be8bc2-5d53-49d4-8ae0-cd281d8d511c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 01:04:33.202967 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:33.202928    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 01:04:33.203177 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:33.202992    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp" podUID="51be8bc2-5d53-49d4-8ae0-cd281d8d511c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 01:04:34.202538 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:34.202496    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 01:04:34.203079 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:34.202599    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp" podUID="51be8bc2-5d53-49d4-8ae0-cd281d8d511c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 01:04:35.203325 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:35.203289    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 01:04:35.203887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:35.203340    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp" podUID="51be8bc2-5d53-49d4-8ae0-cd281d8d511c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 01:04:36.203348 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:36.203311    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 01:04:36.203787 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:36.203406    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp" podUID="51be8bc2-5d53-49d4-8ae0-cd281d8d511c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 01:04:37.208755 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:37.208699    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:37.209285 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:37.209261    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:37.210139 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:37.210115    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:04:37.655014 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:37.654981    6099 generic.go:356] "Generic (PLEG): container finished" podID="67037690-1455-44ab-aef7-8adb912e3077" containerID="c5d4c46681067afaec4d5a7b7b938d43b50bba061dd223d8f1259b7e13c9e23f" exitCode=0
Oct 30 01:04:37.655201 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:37.655057    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4" event={"ID":"67037690-1455-44ab-aef7-8adb912e3077","Type":"ContainerDied","Data":"c5d4c46681067afaec4d5a7b7b938d43b50bba061dd223d8f1259b7e13c9e23f"}
Oct 30 01:04:37.655876 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:37.655858    6099 scope.go:122] "RemoveContainer" containerID="c5d4c46681067afaec4d5a7b7b938d43b50bba061dd223d8f1259b7e13c9e23f"
Oct 30 01:04:37.657870 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:37.657834    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:04:37.657987 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:37.657924    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:04:37.657987 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:37.657972    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:04:38.691881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:38.691845    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4" event={"ID":"67037690-1455-44ab-aef7-8adb912e3077","Type":"ContainerStarted","Data":"805abbd20a47897e469ce77c73c514d064348ddb338cb88292f7f9a990d8741c"}
Oct 30 01:04:39.508289 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:39.508196    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:04:39.508289 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:39.508232    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 01:04:39.508566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:39.508295    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4" podUID="67037690-1455-44ab-aef7-8adb912e3077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 01:04:39.703606 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:39.703573    6099 generic.go:356] "Generic (PLEG): container finished" podID="cb207b92-e133-428b-99b2-e17d2e903511" containerID="3f9aa1642154abcfdac8ab90306762bc06602008b969c02c9c6bb449a6320bb7" exitCode=0
Oct 30 01:04:39.704056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:39.703649    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5" event={"ID":"cb207b92-e133-428b-99b2-e17d2e903511","Type":"ContainerDied","Data":"3f9aa1642154abcfdac8ab90306762bc06602008b969c02c9c6bb449a6320bb7"}
Oct 30 01:04:39.704227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:39.704209    6099 scope.go:122] "RemoveContainer" containerID="3f9aa1642154abcfdac8ab90306762bc06602008b969c02c9c6bb449a6320bb7"
Oct 30 01:04:39.706447 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:39.706417    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:04:39.706542 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:39.706485    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:04:39.706542 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:39.706522    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:04:40.512732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:40.511371    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 01:04:40.512732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:40.511453    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4" podUID="67037690-1455-44ab-aef7-8adb912e3077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 01:04:40.728097 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:40.728057    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5" event={"ID":"cb207b92-e133-428b-99b2-e17d2e903511","Type":"ContainerStarted","Data":"ca9799fa667b3812e1d8a87d11a2f76fc5059a155ccdf9d0082145959358afaf"}
Oct 30 01:04:41.187730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:41.184210    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:04:41.187730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:41.185093    6099 patch_prober.go:29] interesting pod/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 30 01:04:41.187730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:41.185151    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5" podUID="cb207b92-e133-428b-99b2-e17d2e903511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 30 01:04:41.513820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:41.512077    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 01:04:41.513820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:41.512151    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4" podUID="67037690-1455-44ab-aef7-8adb912e3077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 01:04:42.190729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:42.190128    6099 patch_prober.go:29] interesting pod/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 30 01:04:42.190729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:42.190207    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5" podUID="cb207b92-e133-428b-99b2-e17d2e903511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 30 01:04:42.519479 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:42.518871    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 01:04:42.519479 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:42.518928    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4" podUID="67037690-1455-44ab-aef7-8adb912e3077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 01:04:43.212729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:43.209804    6099 patch_prober.go:29] interesting pod/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 30 01:04:43.212729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:43.209865    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5" podUID="cb207b92-e133-428b-99b2-e17d2e903511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 30 01:04:43.514365 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:43.510154    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 30 01:04:43.514365 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:43.510223    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4" podUID="67037690-1455-44ab-aef7-8adb912e3077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 30 01:04:44.154269 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:44.154236    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:04:44.196350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:44.196315    6099 patch_prober.go:29] interesting pod/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 30 01:04:44.196649 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:44.196620    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5" podUID="cb207b92-e133-428b-99b2-e17d2e903511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 30 01:04:44.512622 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:44.512538    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:04:44.513027 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:44.512898    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:04:44.513518 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:44.513493    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:04:45.190732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:45.190193    6099 patch_prober.go:29] interesting pod/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 30 01:04:45.190732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:45.190251    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5" podUID="cb207b92-e133-428b-99b2-e17d2e903511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 30 01:04:46.183577 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:46.183544    6099 patch_prober.go:29] interesting pod/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 30 01:04:46.184102 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:46.184072    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5" podUID="cb207b92-e133-428b-99b2-e17d2e903511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 30 01:04:46.631985 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:04:46.631952    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:04:47.188808 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:47.188779    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:04:47.189907 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:04:47.189887    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:05:06.622094 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:06.622063    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 01:05:06.647348 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:06.647318    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 01:05:11.552657 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:11.552617    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:05:11.920133 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:11.920052    6099 scope.go:122] "RemoveContainer" containerID="38eef8fe5fc431b0e422dc7441cfb66179478c333750132ed2c0970eeb0da690"
Oct 30 01:05:11.933525 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:11.933407    6099 scope.go:122] "RemoveContainer" containerID="f5c1d7bffa355474fb32e673091be70baf903e4d960c3bbcd5e7435d4a72b430"
Oct 30 01:05:11.945778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:11.945757    6099 scope.go:122] "RemoveContainer" containerID="1d16df1e835e2d935cffd7f1de1d5dabcaa306c09d12ce2975002d795dd9c331"
Oct 30 01:05:11.957431 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:11.957408    6099 scope.go:122] "RemoveContainer" containerID="f34d797730927935a2ca15215509d10cc28faeb785d849e25284c35818ca9082"
Oct 30 01:05:11.969047 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:11.969023    6099 scope.go:122] "RemoveContainer" containerID="dabf41fbd737ba48c1143200928866ccd5ea3f82d5430d2b797778c3668f4b92"
Oct 30 01:05:13.036904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.036850    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"]
Oct 30 01:05:13.068061 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.068030    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.095749 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.093281    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"]
Oct 30 01:05:13.116588 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.116549    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c5c9922-e50a-47a1-8b74-021582765920-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.116933 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.116910    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.117144 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.117126    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.117316 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.117300    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.117429 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.117414    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c5c9922-e50a-47a1-8b74-021582765920-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.117577 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.117562    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6564\" (UniqueName: \"kubernetes.io/projected/1c5c9922-e50a-47a1-8b74-021582765920-kube-api-access-f6564\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.117903 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.117883    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c5c9922-e50a-47a1-8b74-021582765920-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.118045 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.118029    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.118158 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.118143    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.222731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.220296    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c5c9922-e50a-47a1-8b74-021582765920-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.222731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.220358    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.222731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.220391    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.222731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.220511    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c5c9922-e50a-47a1-8b74-021582765920-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.222731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.220602    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.222731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.220686    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.222731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.220811    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.222731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.220839    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c5c9922-e50a-47a1-8b74-021582765920-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.222731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.220911    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6564\" (UniqueName: \"kubernetes.io/projected/1c5c9922-e50a-47a1-8b74-021582765920-kube-api-access-f6564\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.222731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.222126    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-workload-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.223618 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.223526    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-istio-data\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.224116 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.224025    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-credential-socket\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.224800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.224779    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-workload-certs\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.225084 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.225063    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c5c9922-e50a-47a1-8b74-021582765920-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.232110 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.232077    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-istio-envoy\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.237381 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.237357    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c5c9922-e50a-47a1-8b74-021582765920-istio-podinfo\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.253128 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.253093    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c5c9922-e50a-47a1-8b74-021582765920-istio-token\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.256017 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.255117    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6564\" (UniqueName: \"kubernetes.io/projected/1c5c9922-e50a-47a1-8b74-021582765920-kube-api-access-f6564\") pod \"e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") " pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.441251 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.441214    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:13.831729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.819724    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"]
Oct 30 01:05:13.878749 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:05:13.873744    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5c9922_e50a_47a1_8b74_021582765920.slice/crio-33b156e01376bdcaeed0fdf7e6d45bcadb7169d72da64eeb7e664e21e6c8c103 WatchSource:0}: Error finding container 33b156e01376bdcaeed0fdf7e6d45bcadb7169d72da64eeb7e664e21e6c8c103: Status 404 returned error can't find the container with id 33b156e01376bdcaeed0fdf7e6d45bcadb7169d72da64eeb7e664e21e6c8c103
Oct 30 01:05:13.911057 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.910161    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:05:13.911057 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.910266    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:05:13.911057 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:13.910312    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:05:14.113215 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:14.102696    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f" event={"ID":"1c5c9922-e50a-47a1-8b74-021582765920","Type":"ContainerStarted","Data":"33b156e01376bdcaeed0fdf7e6d45bcadb7169d72da64eeb7e664e21e6c8c103"}
Oct 30 01:05:15.139932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:15.139897    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f" event={"ID":"1c5c9922-e50a-47a1-8b74-021582765920","Type":"ContainerStarted","Data":"b09e931d70350390da13f5134a9ea4c65a6798eb98466bf4173cd5e5e80facd7"}
Oct 30 01:05:15.238416 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:15.235657    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f" podStartSLOduration=3.235635589 podStartE2EDuration="3.235635589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:15.230456122 +0000 UTC m=+1809.087363878" watchObservedRunningTime="2026-10-30 01:05:15.235635589 +0000 UTC m=+1809.092543345"
Oct 30 01:05:15.442847 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:15.442758    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:15.444319 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:15.444295    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 30 01:05:15.444441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:15.444349    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f" podUID="1c5c9922-e50a-47a1-8b74-021582765920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 30 01:05:16.442590 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:16.442555    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 30 01:05:16.443011 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:16.442629    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f" podUID="1c5c9922-e50a-47a1-8b74-021582765920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 30 01:05:16.633682 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:05:16.633634    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:05:17.442897 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:17.442855    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 30 01:05:17.443376 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:17.442917    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f" podUID="1c5c9922-e50a-47a1-8b74-021582765920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 30 01:05:18.442507 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:18.442424    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 30 01:05:18.442507 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:18.442481    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f" podUID="1c5c9922-e50a-47a1-8b74-021582765920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 30 01:05:19.441867 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:19.441831    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 30 01:05:19.442360 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:19.441898    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f" podUID="1c5c9922-e50a-47a1-8b74-021582765920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 30 01:05:20.442053 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:20.442022    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 30 01:05:20.442469 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:20.442073    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f" podUID="1c5c9922-e50a-47a1-8b74-021582765920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 30 01:05:21.447229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:21.447196    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:21.447720 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:21.447647    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:21.448395 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:21.448373    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:05:46.632408 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:05:46.632369    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:05:53.036884 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:53.036847    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26"]
Oct 30 01:05:53.048258 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:53.048225    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866af8b6-2xs26"]
Oct 30 01:05:54.264769 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:05:54.264736    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b77c32-727e-41e1-9303-5262f1b1e3b0" path="/var/lib/kubelet/pods/b6b77c32-727e-41e1-9303-5262f1b1e3b0/volumes"
Oct 30 01:06:06.418261 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:06.418210    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"]
Oct 30 01:06:06.419988 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:06.419935    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z" podUID="4be8330a-cd36-47fa-b660-05a5796f3473" containerName="istio-proxy" containerID="cri-o://1c6f5888e46c6bc2fca989973604ab88689290376538279d8732173f48f3a58b" gracePeriod=30
Oct 30 01:06:11.842481 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:11.842445    6099 generic.go:356] "Generic (PLEG): container finished" podID="4be8330a-cd36-47fa-b660-05a5796f3473" containerID="1c6f5888e46c6bc2fca989973604ab88689290376538279d8732173f48f3a58b" exitCode=0
Oct 30 01:06:11.848415 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:11.848370    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z" event={"ID":"4be8330a-cd36-47fa-b660-05a5796f3473","Type":"ContainerDied","Data":"1c6f5888e46c6bc2fca989973604ab88689290376538279d8732173f48f3a58b"}
Oct 30 01:06:11.998727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:11.996158    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 01:06:12.035319 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.035279    6099 scope.go:122] "RemoveContainer" containerID="37719d474c5121bf1ea1a50bd06a617d6923f043cfadc32cf496938fffe1f934"
Oct 30 01:06:12.061606 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.061584    6099 scope.go:122] "RemoveContainer" containerID="f0f6d35d6eecd514dcc639efc33a948bf9246a7b6e63c7207e57f9e6fa408ab3"
Oct 30 01:06:12.062291 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.062258    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"]
Oct 30 01:06:12.063690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.063659    6099 cpu_manager.go:414] "containerMap: removing container" podUID="4be8330a-cd36-47fa-b660-05a5796f3473" containerName="istio-proxy"
Oct 30 01:06:12.063690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.063684    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be8330a-cd36-47fa-b660-05a5796f3473" containerName="istio-proxy"
Oct 30 01:06:12.064006 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.063988    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be8330a-cd36-47fa-b660-05a5796f3473" containerName="istio-proxy"
Oct 30 01:06:12.078736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.078350    6099 scope.go:122] "RemoveContainer" containerID="d81c7c210ad018cbdbb899188b2f9e1dedd47360ac4abd32c85c1a05816b46cd"
Oct 30 01:06:12.085592 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.085328    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.099984 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.099937    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"]
Oct 30 01:06:12.111840 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.110543    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be8330a-cd36-47fa-b660-05a5796f3473-istio-token\" (UniqueName: \"kubernetes.io/projected/4be8330a-cd36-47fa-b660-05a5796f3473-istio-token\") pod \"4be8330a-cd36-47fa-b660-05a5796f3473\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") "
Oct 30 01:06:12.111840 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.110896    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-istio-data\") pod \"4be8330a-cd36-47fa-b660-05a5796f3473\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") "
Oct 30 01:06:12.111840 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.110979    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4be8330a-cd36-47fa-b660-05a5796f3473-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be8330a-cd36-47fa-b660-05a5796f3473-istio-podinfo\") pod \"4be8330a-cd36-47fa-b660-05a5796f3473\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") "
Oct 30 01:06:12.111840 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.111129    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-workload-socket\") pod \"4be8330a-cd36-47fa-b660-05a5796f3473\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") "
Oct 30 01:06:12.111840 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.111200    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-istio-envoy\") pod \"4be8330a-cd36-47fa-b660-05a5796f3473\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") "
Oct 30 01:06:12.111840 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.111266    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-workload-certs\") pod \"4be8330a-cd36-47fa-b660-05a5796f3473\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") "
Oct 30 01:06:12.111840 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.111432    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-credential-socket\") pod \"4be8330a-cd36-47fa-b660-05a5796f3473\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") "
Oct 30 01:06:12.111840 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.111531    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be8330a-cd36-47fa-b660-05a5796f3473-kube-api-access-4srvh\" (UniqueName: \"kubernetes.io/projected/4be8330a-cd36-47fa-b660-05a5796f3473-kube-api-access-4srvh\") pod \"4be8330a-cd36-47fa-b660-05a5796f3473\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") "
Oct 30 01:06:12.111840 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.111667    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be8330a-cd36-47fa-b660-05a5796f3473-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be8330a-cd36-47fa-b660-05a5796f3473-istiod-ca-cert\") pod \"4be8330a-cd36-47fa-b660-05a5796f3473\" (UID: \"4be8330a-cd36-47fa-b660-05a5796f3473\") "
Oct 30 01:06:12.112366 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.112120    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-credential-socket" pod "4be8330a-cd36-47fa-b660-05a5796f3473" (UID: "4be8330a-cd36-47fa-b660-05a5796f3473"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:12.113010 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.112981    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-workload-socket" pod "4be8330a-cd36-47fa-b660-05a5796f3473" (UID: "4be8330a-cd36-47fa-b660-05a5796f3473"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:12.113204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.113178    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be8330a-cd36-47fa-b660-05a5796f3473-istiod-ca-cert" pod "4be8330a-cd36-47fa-b660-05a5796f3473" (UID: "4be8330a-cd36-47fa-b660-05a5796f3473"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:12.113284 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.113262    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:12.113342 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.113284    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:12.113397 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.113362    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-istio-data" pod "4be8330a-cd36-47fa-b660-05a5796f3473" (UID: "4be8330a-cd36-47fa-b660-05a5796f3473"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:12.113666 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.113643    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-workload-certs" pod "4be8330a-cd36-47fa-b660-05a5796f3473" (UID: "4be8330a-cd36-47fa-b660-05a5796f3473"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:12.116323 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.116300    6099 scope.go:122] "RemoveContainer" containerID="1c6f5888e46c6bc2fca989973604ab88689290376538279d8732173f48f3a58b"
Oct 30 01:06:12.118317 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.118281    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be8330a-cd36-47fa-b660-05a5796f3473-kube-api-access-4srvh" pod "4be8330a-cd36-47fa-b660-05a5796f3473" (UID: "4be8330a-cd36-47fa-b660-05a5796f3473"). InnerVolumeSpecName "kube-api-access-4srvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:12.119161 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.118609    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-istio-envoy" pod "4be8330a-cd36-47fa-b660-05a5796f3473" (UID: "4be8330a-cd36-47fa-b660-05a5796f3473"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:12.119161 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.118726    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4be8330a-cd36-47fa-b660-05a5796f3473-istio-podinfo" pod "4be8330a-cd36-47fa-b660-05a5796f3473" (UID: "4be8330a-cd36-47fa-b660-05a5796f3473"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:06:12.128929 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.128896    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be8330a-cd36-47fa-b660-05a5796f3473-istio-token" pod "4be8330a-cd36-47fa-b660-05a5796f3473" (UID: "4be8330a-cd36-47fa-b660-05a5796f3473"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:12.227730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.226615    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-workload-certs\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.227730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.226678    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-credential-socket\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.227730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.226874    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f728e81-7497-41de-874a-95b05a1c7395-istio-podinfo\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.227730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.226992    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-istio-envoy\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.227730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.227028    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nvlz\" (UniqueName: \"kubernetes.io/projected/3f728e81-7497-41de-874a-95b05a1c7395-kube-api-access-4nvlz\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.227730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.227058    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-istio-data\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.227730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.227174    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f728e81-7497-41de-874a-95b05a1c7395-istio-token\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.227730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.227204    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-workload-socket\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.227730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.227319    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f728e81-7497-41de-874a-95b05a1c7395-istiod-ca-cert\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.227730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.227514    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:12.227730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.227532    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be8330a-cd36-47fa-b660-05a5796f3473-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:12.227730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.227546    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:12.227730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.227564    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be8330a-cd36-47fa-b660-05a5796f3473-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:12.227730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.227581    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4srvh\" (UniqueName: \"kubernetes.io/projected/4be8330a-cd36-47fa-b660-05a5796f3473-kube-api-access-4srvh\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:12.227730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.227601    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be8330a-cd36-47fa-b660-05a5796f3473-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:12.227730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.227618    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be8330a-cd36-47fa-b660-05a5796f3473-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:12.387783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.381689    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f728e81-7497-41de-874a-95b05a1c7395-istiod-ca-cert\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.387783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.381851    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-workload-certs\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.387783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.381895    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-credential-socket\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.387783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.382055    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f728e81-7497-41de-874a-95b05a1c7395-istio-podinfo\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.387783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.382213    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-istio-envoy\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.387783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.382244    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nvlz\" (UniqueName: \"kubernetes.io/projected/3f728e81-7497-41de-874a-95b05a1c7395-kube-api-access-4nvlz\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.387783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.382273    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-istio-data\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.387783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.382386    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f728e81-7497-41de-874a-95b05a1c7395-istio-token\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.387783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.382412    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-workload-socket\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.387783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.382964    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-workload-socket\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.557800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.424257    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-istio-data\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.557800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.424523    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-workload-certs\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.557800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.424813    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-credential-socket\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.557800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.424965    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f728e81-7497-41de-874a-95b05a1c7395-istiod-ca-cert\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.604392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.604340    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f728e81-7497-41de-874a-95b05a1c7395-istio-podinfo\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.674517 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.674416    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-istio-envoy\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.719165 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.718997    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nvlz\" (UniqueName: \"kubernetes.io/projected/3f728e81-7497-41de-874a-95b05a1c7395-kube-api-access-4nvlz\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.734532 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.733382    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f728e81-7497-41de-874a-95b05a1c7395-istio-token\") pod \"e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") " pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:12.880474 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.880426    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z" event={"ID":"4be8330a-cd36-47fa-b660-05a5796f3473","Type":"ContainerDied","Data":"56384f77eeea2c4e436564f510bf4878eb3186f4077b037d7587b3ccee09062f"}
Oct 30 01:06:12.881182 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.881156    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"
Oct 30 01:06:12.994292 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:12.994199    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"]
Oct 30 01:06:13.005088 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:13.005053    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-ndw8z"]
Oct 30 01:06:13.025745 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:13.024489    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:13.572529 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:06:13.572499    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f728e81_7497_41de_874a_95b05a1c7395.slice/crio-304c9a6888e0edcc950d13c42864c5840ef5dca2ca9b43a92e87255136609610 WatchSource:0}: Error finding container 304c9a6888e0edcc950d13c42864c5840ef5dca2ca9b43a92e87255136609610: Status 404 returned error can't find the container with id 304c9a6888e0edcc950d13c42864c5840ef5dca2ca9b43a92e87255136609610
Oct 30 01:06:13.573750 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:13.573724    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"]
Oct 30 01:06:13.575497 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:13.575460    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:06:13.575596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:13.575551    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:06:13.575663 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:13.575601    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:06:13.894488 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:13.894396    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq" event={"ID":"3f728e81-7497-41de-874a-95b05a1c7395","Type":"ContainerStarted","Data":"5c31f514f04639615ca9fc5c33db573c6aa4a91d5ac9ba908608ba463e0b9be6"}
Oct 30 01:06:13.894488 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:13.894453    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq" event={"ID":"3f728e81-7497-41de-874a-95b05a1c7395","Type":"ContainerStarted","Data":"304c9a6888e0edcc950d13c42864c5840ef5dca2ca9b43a92e87255136609610"}
Oct 30 01:06:13.943796 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:13.943690    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq" podStartSLOduration=26.943670974 podStartE2EDuration="26.943670974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:13.940005881 +0000 UTC m=+1867.796913649" watchObservedRunningTime="2026-10-30 01:06:13.943670974 +0000 UTC m=+1867.800578731"
Oct 30 01:06:14.025106 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:14.025071    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:14.026445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:14.026416    6099 patch_prober.go:29] interesting pod/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Oct 30 01:06:14.026753 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:14.026482    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq" podUID="3f728e81-7497-41de-874a-95b05a1c7395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Oct 30 01:06:14.289796 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:14.265417    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4be8330a-cd36-47fa-b660-05a5796f3473" path="/var/lib/kubelet/pods/4be8330a-cd36-47fa-b660-05a5796f3473/volumes"
Oct 30 01:06:15.025607 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:15.025570    6099 patch_prober.go:29] interesting pod/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Oct 30 01:06:15.026071 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:15.025622    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq" podUID="3f728e81-7497-41de-874a-95b05a1c7395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Oct 30 01:06:16.024976 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:16.024942    6099 patch_prober.go:29] interesting pod/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Oct 30 01:06:16.025175 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:16.024997    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq" podUID="3f728e81-7497-41de-874a-95b05a1c7395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Oct 30 01:06:16.631246 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:06:16.631208    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:06:17.025466 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:17.025381    6099 patch_prober.go:29] interesting pod/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Oct 30 01:06:17.025466 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:17.025433    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq" podUID="3f728e81-7497-41de-874a-95b05a1c7395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Oct 30 01:06:18.025117 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:18.025078    6099 patch_prober.go:29] interesting pod/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Oct 30 01:06:18.025530 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:18.025155    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq" podUID="3f728e81-7497-41de-874a-95b05a1c7395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Oct 30 01:06:19.025049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:19.025008    6099 patch_prober.go:29] interesting pod/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Oct 30 01:06:19.025289 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:19.025085    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq" podUID="3f728e81-7497-41de-874a-95b05a1c7395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Oct 30 01:06:20.172461 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:20.172423    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:20.173039 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:20.172985    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:20.174758 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:20.174688    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:06:39.009157 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:39.009119    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"]
Oct 30 01:06:39.009613 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:39.009311    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj" podUID="744aef66-2b84-4c2b-951e-97d85260e902" containerName="istio-proxy" containerID="cri-o://52151f66ae44ff27f152af16b296853620b4c65214e5e2d4058d3a6a52a765ae" gracePeriod=30
Oct 30 01:06:39.023021 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:39.022992    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"]
Oct 30 01:06:39.023204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:39.023172    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h" podUID="e351b3f4-02ba-48f9-8298-77e6aa2d148e" containerName="istio-proxy" containerID="cri-o://b4e2bf26a0781bb5e18efa499e132694bcc98d15025c92cb14c54455f6f08731" gracePeriod=30
Oct 30 01:06:39.028991 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:39.028959    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"]
Oct 30 01:06:39.029188 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:39.029166    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm" podUID="04f57545-b25c-45ac-9fd5-bc9c6e1cf068" containerName="istio-proxy" containerID="cri-o://b802d4d73dd948541c74cd0ca0ba2290c3c4fa2efe1b1f80de14208ac2da582e" gracePeriod=30
Oct 30 01:06:44.203255 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.203217    6099 generic.go:356] "Generic (PLEG): container finished" podID="04f57545-b25c-45ac-9fd5-bc9c6e1cf068" containerID="b802d4d73dd948541c74cd0ca0ba2290c3c4fa2efe1b1f80de14208ac2da582e" exitCode=0
Oct 30 01:06:44.203925 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.203261    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm" event={"ID":"04f57545-b25c-45ac-9fd5-bc9c6e1cf068","Type":"ContainerDied","Data":"b802d4d73dd948541c74cd0ca0ba2290c3c4fa2efe1b1f80de14208ac2da582e"}
Oct 30 01:06:44.205941 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.205920    6099 generic.go:356] "Generic (PLEG): container finished" podID="744aef66-2b84-4c2b-951e-97d85260e902" containerID="52151f66ae44ff27f152af16b296853620b4c65214e5e2d4058d3a6a52a765ae" exitCode=0
Oct 30 01:06:44.206020 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.205996    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj" event={"ID":"744aef66-2b84-4c2b-951e-97d85260e902","Type":"ContainerDied","Data":"52151f66ae44ff27f152af16b296853620b4c65214e5e2d4058d3a6a52a765ae"}
Oct 30 01:06:44.208155 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.208133    6099 generic.go:356] "Generic (PLEG): container finished" podID="e351b3f4-02ba-48f9-8298-77e6aa2d148e" containerID="b4e2bf26a0781bb5e18efa499e132694bcc98d15025c92cb14c54455f6f08731" exitCode=0
Oct 30 01:06:44.208293 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.208278    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h" event={"ID":"e351b3f4-02ba-48f9-8298-77e6aa2d148e","Type":"ContainerDied","Data":"b4e2bf26a0781bb5e18efa499e132694bcc98d15025c92cb14c54455f6f08731"}
Oct 30 01:06:44.364246 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.364213    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 01:06:44.401445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.401410    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 01:06:44.404578 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.404554    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 01:06:44.500748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.500698    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-kube-api-access-plnvp\" (UniqueName: \"kubernetes.io/projected/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-kube-api-access-plnvp\") pod \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") "
Oct 30 01:06:44.501015 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.500761    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-credential-socket\") pod \"744aef66-2b84-4c2b-951e-97d85260e902\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") "
Oct 30 01:06:44.501108 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.501047    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-credential-socket" pod "744aef66-2b84-4c2b-951e-97d85260e902" (UID: "744aef66-2b84-4c2b-951e-97d85260e902"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:44.501183 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.501163    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-credential-socket\") pod \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") "
Oct 30 01:06:44.501246 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.501216    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e351b3f4-02ba-48f9-8298-77e6aa2d148e-kube-api-access-ndzcl\" (UniqueName: \"kubernetes.io/projected/e351b3f4-02ba-48f9-8298-77e6aa2d148e-kube-api-access-ndzcl\") pod \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") "
Oct 30 01:06:44.501306 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.501263    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-envoy\") pod \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") "
Oct 30 01:06:44.501368 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.501307    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-istio-envoy\") pod \"744aef66-2b84-4c2b-951e-97d85260e902\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") "
Oct 30 01:06:44.501368 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.501346    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-workload-certs\") pod \"744aef66-2b84-4c2b-951e-97d85260e902\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") "
Oct 30 01:06:44.501480 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.501401    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/744aef66-2b84-4c2b-951e-97d85260e902-kube-api-access-hc99j\" (UniqueName: \"kubernetes.io/projected/744aef66-2b84-4c2b-951e-97d85260e902-kube-api-access-hc99j\") pod \"744aef66-2b84-4c2b-951e-97d85260e902\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") "
Oct 30 01:06:44.501480 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.501432    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/744aef66-2b84-4c2b-951e-97d85260e902-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744aef66-2b84-4c2b-951e-97d85260e902-istio-podinfo\") pod \"744aef66-2b84-4c2b-951e-97d85260e902\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") "
Oct 30 01:06:44.501480 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.501462    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-workload-socket\") pod \"744aef66-2b84-4c2b-951e-97d85260e902\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") "
Oct 30 01:06:44.501638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.501491    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-workload-socket\") pod \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") "
Oct 30 01:06:44.501638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.501565    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-envoy\") pod \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") "
Oct 30 01:06:44.501638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.501602    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-istio-data\") pod \"744aef66-2b84-4c2b-951e-97d85260e902\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") "
Oct 30 01:06:44.501815 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.501742    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-data\") pod \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") "
Oct 30 01:06:44.501888 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.501814    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-workload-certs\") pod \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") "
Oct 30 01:06:44.501888 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.501841    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-workload-certs" pod "744aef66-2b84-4c2b-951e-97d85260e902" (UID: "744aef66-2b84-4c2b-951e-97d85260e902"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:44.502005 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.501903    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/744aef66-2b84-4c2b-951e-97d85260e902-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744aef66-2b84-4c2b-951e-97d85260e902-istiod-ca-cert\") pod \"744aef66-2b84-4c2b-951e-97d85260e902\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") "
Oct 30 01:06:44.502005 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.501947    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-credential-socket\") pod \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") "
Oct 30 01:06:44.502005 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.501988    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-data\") pod \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") "
Oct 30 01:06:44.502165 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.502018    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-workload-socket\") pod \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") "
Oct 30 01:06:44.502165 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.502053    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-token\" (UniqueName: \"kubernetes.io/projected/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-token\") pod \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") "
Oct 30 01:06:44.502165 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.502123    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/744aef66-2b84-4c2b-951e-97d85260e902-istio-token\" (UniqueName: \"kubernetes.io/projected/744aef66-2b84-4c2b-951e-97d85260e902-istio-token\") pod \"744aef66-2b84-4c2b-951e-97d85260e902\" (UID: \"744aef66-2b84-4c2b-951e-97d85260e902\") "
Oct 30 01:06:44.502321 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.502163    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-token\" (UniqueName: \"kubernetes.io/projected/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-token\") pod \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") "
Oct 30 01:06:44.502321 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.502200    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-workload-certs\") pod \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") "
Oct 30 01:06:44.502321 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.502225    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-credential-socket" pod "04f57545-b25c-45ac-9fd5-bc9c6e1cf068" (UID: "04f57545-b25c-45ac-9fd5-bc9c6e1cf068"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:44.502321 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.502262    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istiod-ca-cert\") pod \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") "
Oct 30 01:06:44.502531 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.502397    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-workload-certs" pod "e351b3f4-02ba-48f9-8298-77e6aa2d148e" (UID: "e351b3f4-02ba-48f9-8298-77e6aa2d148e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:44.503905 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.502759    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744aef66-2b84-4c2b-951e-97d85260e902-istiod-ca-cert" pod "744aef66-2b84-4c2b-951e-97d85260e902" (UID: "744aef66-2b84-4c2b-951e-97d85260e902"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:44.503905 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.503118    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-data" pod "04f57545-b25c-45ac-9fd5-bc9c6e1cf068" (UID: "04f57545-b25c-45ac-9fd5-bc9c6e1cf068"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:44.503905 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.503319    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-credential-socket" pod "e351b3f4-02ba-48f9-8298-77e6aa2d148e" (UID: "e351b3f4-02ba-48f9-8298-77e6aa2d148e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:44.503905 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.503511    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-workload-socket" pod "e351b3f4-02ba-48f9-8298-77e6aa2d148e" (UID: "e351b3f4-02ba-48f9-8298-77e6aa2d148e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:44.504228 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.503986    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istiod-ca-cert" pod "04f57545-b25c-45ac-9fd5-bc9c6e1cf068" (UID: "04f57545-b25c-45ac-9fd5-bc9c6e1cf068"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:44.504352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.504325    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-workload-socket" pod "04f57545-b25c-45ac-9fd5-bc9c6e1cf068" (UID: "04f57545-b25c-45ac-9fd5-bc9c6e1cf068"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:44.504441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.504411    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-istio-data" pod "744aef66-2b84-4c2b-951e-97d85260e902" (UID: "744aef66-2b84-4c2b-951e-97d85260e902"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:44.504581 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.504453    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-workload-certs" pod "04f57545-b25c-45ac-9fd5-bc9c6e1cf068" (UID: "04f57545-b25c-45ac-9fd5-bc9c6e1cf068"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:44.505121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.505084    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-podinfo\") pod \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\" (UID: \"04f57545-b25c-45ac-9fd5-bc9c6e1cf068\") "
Oct 30 01:06:44.505244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.505209    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-podinfo\") pod \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") "
Oct 30 01:06:44.505315 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.505288    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istiod-ca-cert\") pod \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\" (UID: \"e351b3f4-02ba-48f9-8298-77e6aa2d148e\") "
Oct 30 01:06:44.507220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.507193    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.507411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.507225    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.507411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.507240    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.507411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.507254    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.507411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.507268    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.507411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.507281    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.507411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.507295    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744aef66-2b84-4c2b-951e-97d85260e902-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.507411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.507307    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.507411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.507320    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.507411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.507334    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.507411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.507346    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.507411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.507359    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.508185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.507608    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-data" pod "e351b3f4-02ba-48f9-8298-77e6aa2d148e" (UID: "e351b3f4-02ba-48f9-8298-77e6aa2d148e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:44.508185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.507672    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-workload-socket" pod "744aef66-2b84-4c2b-951e-97d85260e902" (UID: "744aef66-2b84-4c2b-951e-97d85260e902"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:44.508185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.507892    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istiod-ca-cert" pod "e351b3f4-02ba-48f9-8298-77e6aa2d148e" (UID: "e351b3f4-02ba-48f9-8298-77e6aa2d148e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:44.508185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.508106    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744aef66-2b84-4c2b-951e-97d85260e902-istio-token" pod "744aef66-2b84-4c2b-951e-97d85260e902" (UID: "744aef66-2b84-4c2b-951e-97d85260e902"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:44.508466 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.508387    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-podinfo" pod "e351b3f4-02ba-48f9-8298-77e6aa2d148e" (UID: "e351b3f4-02ba-48f9-8298-77e6aa2d148e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:06:44.511368 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.511342    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-podinfo" pod "04f57545-b25c-45ac-9fd5-bc9c6e1cf068" (UID: "04f57545-b25c-45ac-9fd5-bc9c6e1cf068"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:06:44.511482 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.511416    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-kube-api-access-plnvp" pod "04f57545-b25c-45ac-9fd5-bc9c6e1cf068" (UID: "04f57545-b25c-45ac-9fd5-bc9c6e1cf068"). InnerVolumeSpecName "kube-api-access-plnvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:44.515741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.515694    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/744aef66-2b84-4c2b-951e-97d85260e902-istio-podinfo" pod "744aef66-2b84-4c2b-951e-97d85260e902" (UID: "744aef66-2b84-4c2b-951e-97d85260e902"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:06:44.516109 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.515996    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-token" pod "e351b3f4-02ba-48f9-8298-77e6aa2d148e" (UID: "e351b3f4-02ba-48f9-8298-77e6aa2d148e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:44.516109 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.516048    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744aef66-2b84-4c2b-951e-97d85260e902-kube-api-access-hc99j" pod "744aef66-2b84-4c2b-951e-97d85260e902" (UID: "744aef66-2b84-4c2b-951e-97d85260e902"). InnerVolumeSpecName "kube-api-access-hc99j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:44.516109 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.516076    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-istio-envoy" pod "744aef66-2b84-4c2b-951e-97d85260e902" (UID: "744aef66-2b84-4c2b-951e-97d85260e902"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:44.516431 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.516401    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e351b3f4-02ba-48f9-8298-77e6aa2d148e-kube-api-access-ndzcl" pod "e351b3f4-02ba-48f9-8298-77e6aa2d148e" (UID: "e351b3f4-02ba-48f9-8298-77e6aa2d148e"). InnerVolumeSpecName "kube-api-access-ndzcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:44.516431 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.516414    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-envoy" pod "e351b3f4-02ba-48f9-8298-77e6aa2d148e" (UID: "e351b3f4-02ba-48f9-8298-77e6aa2d148e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:44.516657 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.516635    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-token" pod "04f57545-b25c-45ac-9fd5-bc9c6e1cf068" (UID: "04f57545-b25c-45ac-9fd5-bc9c6e1cf068"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:44.518680 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.518645    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-envoy" pod "04f57545-b25c-45ac-9fd5-bc9c6e1cf068" (UID: "04f57545-b25c-45ac-9fd5-bc9c6e1cf068"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:44.609062 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.608977    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.609062 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.609012    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.609062 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.609035    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.609062 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.609044    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plnvp\" (UniqueName: \"kubernetes.io/projected/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-kube-api-access-plnvp\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.609062 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.609055    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndzcl\" (UniqueName: \"kubernetes.io/projected/e351b3f4-02ba-48f9-8298-77e6aa2d148e-kube-api-access-ndzcl\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.609062 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.609065    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.609392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.609073    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.609392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.609082    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc99j\" (UniqueName: \"kubernetes.io/projected/744aef66-2b84-4c2b-951e-97d85260e902-kube-api-access-hc99j\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.609392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.609090    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744aef66-2b84-4c2b-951e-97d85260e902-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.609392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.609098    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744aef66-2b84-4c2b-951e-97d85260e902-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.609392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.609107    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.609392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.609115    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.609392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.609122    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e351b3f4-02ba-48f9-8298-77e6aa2d148e-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.609392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.609133    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/744aef66-2b84-4c2b-951e-97d85260e902-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:44.609392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:44.609145    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04f57545-b25c-45ac-9fd5-bc9c6e1cf068-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:45.217915 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:45.217883    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"
Oct 30 01:06:45.217915 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:45.217904    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm" event={"ID":"04f57545-b25c-45ac-9fd5-bc9c6e1cf068","Type":"ContainerDied","Data":"a9b159cc7b94d608fe777aaed85758c396433af580fe9802708e6a23a84f7f49"}
Oct 30 01:06:45.218432 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:45.217972    6099 scope.go:122] "RemoveContainer" containerID="b802d4d73dd948541c74cd0ca0ba2290c3c4fa2efe1b1f80de14208ac2da582e"
Oct 30 01:06:45.219772 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:45.219749    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"
Oct 30 01:06:45.219892 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:45.219757    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj" event={"ID":"744aef66-2b84-4c2b-951e-97d85260e902","Type":"ContainerDied","Data":"fc22126a6e8390e9682d7ceb5634d767054994b5480fa73e78b30ae804240909"}
Oct 30 01:06:45.222127 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:45.222097    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h" event={"ID":"e351b3f4-02ba-48f9-8298-77e6aa2d148e","Type":"ContainerDied","Data":"8d91dc400a95fc9cc76e8d7bb660f19980eb089beb9efa46ad20beee150ca581"}
Oct 30 01:06:45.222343 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:45.222321    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"
Oct 30 01:06:45.241469 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:45.241445    6099 scope.go:122] "RemoveContainer" containerID="52151f66ae44ff27f152af16b296853620b4c65214e5e2d4058d3a6a52a765ae"
Oct 30 01:06:45.253640 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:45.253616    6099 scope.go:122] "RemoveContainer" containerID="b4e2bf26a0781bb5e18efa499e132694bcc98d15025c92cb14c54455f6f08731"
Oct 30 01:06:45.298156 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:45.298108    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"]
Oct 30 01:06:45.310170 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:45.310149    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-wzsbm"]
Oct 30 01:06:45.351779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:45.351749    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"]
Oct 30 01:06:45.355497 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:45.355470    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cmb2h"]
Oct 30 01:06:45.389875 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:45.389850    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"]
Oct 30 01:06:45.400918 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:45.400888    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-p4gkj"]
Oct 30 01:06:45.464918 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:45.463810    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn"]
Oct 30 01:06:45.476897 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:45.476867    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h"]
Oct 30 01:06:46.161730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:46.153302    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:06:46.304754 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:46.304699    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04f57545-b25c-45ac-9fd5-bc9c6e1cf068" path="/var/lib/kubelet/pods/04f57545-b25c-45ac-9fd5-bc9c6e1cf068/volumes"
Oct 30 01:06:46.309645 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:46.309602    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="744aef66-2b84-4c2b-951e-97d85260e902" path="/var/lib/kubelet/pods/744aef66-2b84-4c2b-951e-97d85260e902/volumes"
Oct 30 01:06:46.310640 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:46.310618    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e351b3f4-02ba-48f9-8298-77e6aa2d148e" path="/var/lib/kubelet/pods/e351b3f4-02ba-48f9-8298-77e6aa2d148e/volumes"
Oct 30 01:06:46.632165 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:06:46.632130    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:06:49.304730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:49.284724    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:49.457408 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:49.457369    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:06:50.464805 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:50.464682    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn" podUID="c8c2f8c2-32f1-4acb-a67a-44b18bf39464" containerName="payload-processing" containerID="cri-o://fd2435d0df5c386a00f1c8e11f28bc713086893a10380780385228d9a5e33048" gracePeriod=25
Oct 30 01:06:50.477119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:50.477083    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h" podUID="3064087b-a922-4f17-910e-697c6960c3e4" containerName="payload-pre-processing" containerID="cri-o://5c8a879c5d4c5d680d6e96bb067be892b122700709aae19f0b802f186a80b55d" gracePeriod=25
Oct 30 01:06:53.393217 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:53.393181    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"]
Oct 30 01:06:53.393613 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:53.393454    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb" podUID="3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83" containerName="maas-api" containerID="cri-o://852dd853318a644ee6423e99e4a519aa2c276e7a1764069179f1c7b47b0002af" gracePeriod=30
Oct 30 01:06:54.010729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.008108    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"
Oct 30 01:06:54.131157 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.131126    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-metrics-certs\") pod \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\" (UID: \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\") "
Oct 30 01:06:54.131355 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.131197    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-maas-api-tls\") pod \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\" (UID: \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\") "
Oct 30 01:06:54.131484 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.131460    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-kube-api-access-fc8lg\" (UniqueName: \"kubernetes.io/projected/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-kube-api-access-fc8lg\") pod \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\" (UID: \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\") "
Oct 30 01:06:54.131585 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.131514    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-openshift-service-ca\") pod \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\" (UID: \"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83\") "
Oct 30 01:06:54.132130 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.132090    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-openshift-service-ca" pod "3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83" (UID: "3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:54.132658 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.132631    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.133752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.133724    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-maas-api-tls" pod "3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83" (UID: "3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:06:54.133853 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.133832    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-kube-api-access-fc8lg" pod "3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83" (UID: "3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83"). InnerVolumeSpecName "kube-api-access-fc8lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:54.134956 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.134936    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-metrics-certs" pod "3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83" (UID: "3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:06:54.233944 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.233869    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.233944 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.233902    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.233944 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.233911    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc8lg\" (UniqueName: \"kubernetes.io/projected/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83-kube-api-access-fc8lg\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.509313 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.509224    6099 generic.go:356] "Generic (PLEG): container finished" podID="3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83" containerID="852dd853318a644ee6423e99e4a519aa2c276e7a1764069179f1c7b47b0002af" exitCode=0
Oct 30 01:06:54.509313 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.509264    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"
Oct 30 01:06:54.509313 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.509287    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb" event={"ID":"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83","Type":"ContainerDied","Data":"852dd853318a644ee6423e99e4a519aa2c276e7a1764069179f1c7b47b0002af"}
Oct 30 01:06:54.509857 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.509350    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb" event={"ID":"3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83","Type":"ContainerDied","Data":"ccd21283187c09292c6d987b6c59901330c13a4406a878d331e17abb95c45ab7"}
Oct 30 01:06:54.509857 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.509375    6099 scope.go:122] "RemoveContainer" containerID="852dd853318a644ee6423e99e4a519aa2c276e7a1764069179f1c7b47b0002af"
Oct 30 01:06:54.520690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.520663    6099 scope.go:122] "RemoveContainer" containerID="852dd853318a644ee6423e99e4a519aa2c276e7a1764069179f1c7b47b0002af"
Oct 30 01:06:54.521043 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:06:54.521023    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"852dd853318a644ee6423e99e4a519aa2c276e7a1764069179f1c7b47b0002af\": container with ID starting with 852dd853318a644ee6423e99e4a519aa2c276e7a1764069179f1c7b47b0002af not found: ID does not exist" containerID="852dd853318a644ee6423e99e4a519aa2c276e7a1764069179f1c7b47b0002af"
Oct 30 01:06:54.521096 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.521050    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"852dd853318a644ee6423e99e4a519aa2c276e7a1764069179f1c7b47b0002af"} err="failed to get container status \"852dd853318a644ee6423e99e4a519aa2c276e7a1764069179f1c7b47b0002af\": rpc error: code = NotFound desc = could not find container \"852dd853318a644ee6423e99e4a519aa2c276e7a1764069179f1c7b47b0002af\": container with ID starting with 852dd853318a644ee6423e99e4a519aa2c276e7a1764069179f1c7b47b0002af not found: ID does not exist"
Oct 30 01:06:54.584452 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.584410    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"]
Oct 30 01:06:54.590231 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:54.590196    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f4f9b-57f4b65799-ccdgb"]
Oct 30 01:06:56.265621 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:56.265588    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83" path="/var/lib/kubelet/pods/3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83/volumes"
Oct 30 01:06:57.378401 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:57.378369    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn" podUID="c8c2f8c2-32f1-4acb-a67a-44b18bf39464" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:06:57.378401 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:06:57.378394    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h" podUID="3064087b-a922-4f17-910e-697c6960c3e4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:07:01.579494 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:01.579455    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"]
Oct 30 01:07:01.579949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:01.579689    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226" podUID="394e8301-d15d-46d4-aa5c-10f3e154c948" containerName="istio-proxy" containerID="cri-o://2ab1008466a25a41384bc38412922c0c0fb9dc7bdb920a79f11f01d287ada0bb" gracePeriod=30
Oct 30 01:07:06.873569 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.873504    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:07:06.975761 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.975686    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394e8301-d15d-46d4-aa5c-10f3e154c948-kube-api-access-kqblp\" (UniqueName: \"kubernetes.io/projected/394e8301-d15d-46d4-aa5c-10f3e154c948-kube-api-access-kqblp\") pod \"394e8301-d15d-46d4-aa5c-10f3e154c948\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") "
Oct 30 01:07:06.975938 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.975835    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-envoy\") pod \"394e8301-d15d-46d4-aa5c-10f3e154c948\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") "
Oct 30 01:07:06.975938 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.975897    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-credential-socket\") pod \"394e8301-d15d-46d4-aa5c-10f3e154c948\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") "
Oct 30 01:07:06.976076 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.975950    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-token\" (UniqueName: \"kubernetes.io/projected/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-token\") pod \"394e8301-d15d-46d4-aa5c-10f3e154c948\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") "
Oct 30 01:07:06.976147 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.976077    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-workload-certs\") pod \"394e8301-d15d-46d4-aa5c-10f3e154c948\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") "
Oct 30 01:07:06.976205 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.976144    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394e8301-d15d-46d4-aa5c-10f3e154c948-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394e8301-d15d-46d4-aa5c-10f3e154c948-istiod-ca-cert\") pod \"394e8301-d15d-46d4-aa5c-10f3e154c948\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") "
Oct 30 01:07:06.976205 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.976178    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-podinfo\") pod \"394e8301-d15d-46d4-aa5c-10f3e154c948\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") "
Oct 30 01:07:06.976332 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.976239    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-data\") pod \"394e8301-d15d-46d4-aa5c-10f3e154c948\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") "
Oct 30 01:07:06.976332 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.976302    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-workload-socket\") pod \"394e8301-d15d-46d4-aa5c-10f3e154c948\" (UID: \"394e8301-d15d-46d4-aa5c-10f3e154c948\") "
Oct 30 01:07:06.976332 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.976312    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-credential-socket" pod "394e8301-d15d-46d4-aa5c-10f3e154c948" (UID: "394e8301-d15d-46d4-aa5c-10f3e154c948"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:06.976562 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.976536    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-data" pod "394e8301-d15d-46d4-aa5c-10f3e154c948" (UID: "394e8301-d15d-46d4-aa5c-10f3e154c948"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:06.976852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.976830    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-workload-socket" pod "394e8301-d15d-46d4-aa5c-10f3e154c948" (UID: "394e8301-d15d-46d4-aa5c-10f3e154c948"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:06.976963 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.976847    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394e8301-d15d-46d4-aa5c-10f3e154c948-istiod-ca-cert" pod "394e8301-d15d-46d4-aa5c-10f3e154c948" (UID: "394e8301-d15d-46d4-aa5c-10f3e154c948"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:06.977123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.977101    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-workload-certs" pod "394e8301-d15d-46d4-aa5c-10f3e154c948" (UID: "394e8301-d15d-46d4-aa5c-10f3e154c948"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:06.977378 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.977357    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:06.977503 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.977384    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:06.977503 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.977400    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394e8301-d15d-46d4-aa5c-10f3e154c948-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:06.977503 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.977414    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:06.977503 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.977437    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:06.979077 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.979049    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394e8301-d15d-46d4-aa5c-10f3e154c948-kube-api-access-kqblp" pod "394e8301-d15d-46d4-aa5c-10f3e154c948" (UID: "394e8301-d15d-46d4-aa5c-10f3e154c948"). InnerVolumeSpecName "kube-api-access-kqblp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:06.979193 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.979104    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-envoy" pod "394e8301-d15d-46d4-aa5c-10f3e154c948" (UID: "394e8301-d15d-46d4-aa5c-10f3e154c948"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:06.979472 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.979444    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-podinfo" pod "394e8301-d15d-46d4-aa5c-10f3e154c948" (UID: "394e8301-d15d-46d4-aa5c-10f3e154c948"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:07:06.979472 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:06.979458    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-token" pod "394e8301-d15d-46d4-aa5c-10f3e154c948" (UID: "394e8301-d15d-46d4-aa5c-10f3e154c948"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:07.090653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:07.090153    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:07.090653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:07.090194    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:07.090653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:07.090232    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqblp\" (UniqueName: \"kubernetes.io/projected/394e8301-d15d-46d4-aa5c-10f3e154c948-kube-api-access-kqblp\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:07.090653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:07.090251    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394e8301-d15d-46d4-aa5c-10f3e154c948-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:07.378118 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:07.378085    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h" podUID="3064087b-a922-4f17-910e-697c6960c3e4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:07:07.378329 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:07.378082    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn" podUID="c8c2f8c2-32f1-4acb-a67a-44b18bf39464" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:07:07.627216 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:07.627182    6099 generic.go:356] "Generic (PLEG): container finished" podID="394e8301-d15d-46d4-aa5c-10f3e154c948" containerID="2ab1008466a25a41384bc38412922c0c0fb9dc7bdb920a79f11f01d287ada0bb" exitCode=0
Oct 30 01:07:07.627398 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:07.627232    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"
Oct 30 01:07:07.627398 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:07.627272    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226" event={"ID":"394e8301-d15d-46d4-aa5c-10f3e154c948","Type":"ContainerDied","Data":"2ab1008466a25a41384bc38412922c0c0fb9dc7bdb920a79f11f01d287ada0bb"}
Oct 30 01:07:07.627398 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:07.627327    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226" event={"ID":"394e8301-d15d-46d4-aa5c-10f3e154c948","Type":"ContainerDied","Data":"3b24782bdd16ec5dc265f30ba0ba8ae4347c5ab4a5160b99ed1d5c655ef4f9f2"}
Oct 30 01:07:07.627398 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:07.627356    6099 scope.go:122] "RemoveContainer" containerID="2ab1008466a25a41384bc38412922c0c0fb9dc7bdb920a79f11f01d287ada0bb"
Oct 30 01:07:07.642383 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:07.642356    6099 scope.go:122] "RemoveContainer" containerID="2ab1008466a25a41384bc38412922c0c0fb9dc7bdb920a79f11f01d287ada0bb"
Oct 30 01:07:07.642673 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:07:07.642651    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab1008466a25a41384bc38412922c0c0fb9dc7bdb920a79f11f01d287ada0bb\": container with ID starting with 2ab1008466a25a41384bc38412922c0c0fb9dc7bdb920a79f11f01d287ada0bb not found: ID does not exist" containerID="2ab1008466a25a41384bc38412922c0c0fb9dc7bdb920a79f11f01d287ada0bb"
Oct 30 01:07:07.642800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:07.642680    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab1008466a25a41384bc38412922c0c0fb9dc7bdb920a79f11f01d287ada0bb"} err="failed to get container status \"2ab1008466a25a41384bc38412922c0c0fb9dc7bdb920a79f11f01d287ada0bb\": rpc error: code = NotFound desc = could not find container \"2ab1008466a25a41384bc38412922c0c0fb9dc7bdb920a79f11f01d287ada0bb\": container with ID starting with 2ab1008466a25a41384bc38412922c0c0fb9dc7bdb920a79f11f01d287ada0bb not found: ID does not exist"
Oct 30 01:07:07.695384 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:07.695347    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"]
Oct 30 01:07:07.705158 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:07.705128    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5f4f9b-openshift-default-5fbb6b5f5f-lp226"]
Oct 30 01:07:08.264417 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:08.264375    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394e8301-d15d-46d4-aa5c-10f3e154c948" path="/var/lib/kubelet/pods/394e8301-d15d-46d4-aa5c-10f3e154c948/volumes"
Oct 30 01:07:08.470026 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:08.469991    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"]
Oct 30 01:07:08.470223 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:08.470206    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx" podUID="49f719fe-16aa-431d-91b7-1730267521ec" containerName="maas-api" containerID="cri-o://777fabb188f7261f434fae363c14751b7e0e5b89686ce4fbd12fedddb59ee9eb" gracePeriod=30
Oct 30 01:07:08.637209 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:08.637175    6099 generic.go:356] "Generic (PLEG): container finished" podID="49f719fe-16aa-431d-91b7-1730267521ec" containerID="777fabb188f7261f434fae363c14751b7e0e5b89686ce4fbd12fedddb59ee9eb" exitCode=0
Oct 30 01:07:08.637377 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:08.637249    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx" event={"ID":"49f719fe-16aa-431d-91b7-1730267521ec","Type":"ContainerDied","Data":"777fabb188f7261f434fae363c14751b7e0e5b89686ce4fbd12fedddb59ee9eb"}
Oct 30 01:07:08.949960 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:08.949929    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"
Oct 30 01:07:09.057942 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:09.057906    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f719fe-16aa-431d-91b7-1730267521ec-kube-api-access-dmc4m\" (UniqueName: \"kubernetes.io/projected/49f719fe-16aa-431d-91b7-1730267521ec-kube-api-access-dmc4m\") pod \"49f719fe-16aa-431d-91b7-1730267521ec\" (UID: \"49f719fe-16aa-431d-91b7-1730267521ec\") "
Oct 30 01:07:09.072839 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:09.072804    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49f719fe-16aa-431d-91b7-1730267521ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49f719fe-16aa-431d-91b7-1730267521ec-maas-api-tls\") pod \"49f719fe-16aa-431d-91b7-1730267521ec\" (UID: \"49f719fe-16aa-431d-91b7-1730267521ec\") "
Oct 30 01:07:09.073148 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:09.073130    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f719fe-16aa-431d-91b7-1730267521ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f719fe-16aa-431d-91b7-1730267521ec-openshift-service-ca\") pod \"49f719fe-16aa-431d-91b7-1730267521ec\" (UID: \"49f719fe-16aa-431d-91b7-1730267521ec\") "
Oct 30 01:07:09.073325 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:09.073312    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49f719fe-16aa-431d-91b7-1730267521ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/49f719fe-16aa-431d-91b7-1730267521ec-metrics-certs\") pod \"49f719fe-16aa-431d-91b7-1730267521ec\" (UID: \"49f719fe-16aa-431d-91b7-1730267521ec\") "
Oct 30 01:07:09.118159 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:09.107948    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f719fe-16aa-431d-91b7-1730267521ec-openshift-service-ca" pod "49f719fe-16aa-431d-91b7-1730267521ec" (UID: "49f719fe-16aa-431d-91b7-1730267521ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:09.129020 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:09.128679    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f719fe-16aa-431d-91b7-1730267521ec-maas-api-tls" pod "49f719fe-16aa-431d-91b7-1730267521ec" (UID: "49f719fe-16aa-431d-91b7-1730267521ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:07:09.129358 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:09.129325    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f719fe-16aa-431d-91b7-1730267521ec-kube-api-access-dmc4m" pod "49f719fe-16aa-431d-91b7-1730267521ec" (UID: "49f719fe-16aa-431d-91b7-1730267521ec"). InnerVolumeSpecName "kube-api-access-dmc4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:09.131053 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:09.131027    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f719fe-16aa-431d-91b7-1730267521ec-metrics-certs" pod "49f719fe-16aa-431d-91b7-1730267521ec" (UID: "49f719fe-16aa-431d-91b7-1730267521ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:07:09.134475 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:09.134450    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmc4m\" (UniqueName: \"kubernetes.io/projected/49f719fe-16aa-431d-91b7-1730267521ec-kube-api-access-dmc4m\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:09.134682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:09.134667    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49f719fe-16aa-431d-91b7-1730267521ec-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:09.134839 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:09.134825    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f719fe-16aa-431d-91b7-1730267521ec-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:09.134956 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:09.134944    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49f719fe-16aa-431d-91b7-1730267521ec-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:09.647688 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:09.647657    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx" event={"ID":"49f719fe-16aa-431d-91b7-1730267521ec","Type":"ContainerDied","Data":"aa2e14f38fb3be577a959e643daa37061c26445883d48054fb9c28f61b8b05d1"}
Oct 30 01:07:09.648163 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:09.647698    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"
Oct 30 01:07:09.648163 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:09.647723    6099 scope.go:122] "RemoveContainer" containerID="777fabb188f7261f434fae363c14751b7e0e5b89686ce4fbd12fedddb59ee9eb"
Oct 30 01:07:09.704828 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:09.704672    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"]
Oct 30 01:07:09.716479 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:09.716444    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-43c02d-76b4468d97-nwgvx"]
Oct 30 01:07:10.263771 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:10.263735    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49f719fe-16aa-431d-91b7-1730267521ec" path="/var/lib/kubelet/pods/49f719fe-16aa-431d-91b7-1730267521ec/volumes"
Oct 30 01:07:10.665873 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:10.665692    6099 generic.go:356] "Generic (PLEG): container finished" podID="c8c2f8c2-32f1-4acb-a67a-44b18bf39464" containerID="fd2435d0df5c386a00f1c8e11f28bc713086893a10380780385228d9a5e33048" exitCode=0
Oct 30 01:07:10.665873 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:10.665749    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn" event={"ID":"c8c2f8c2-32f1-4acb-a67a-44b18bf39464","Type":"ContainerDied","Data":"fd2435d0df5c386a00f1c8e11f28bc713086893a10380780385228d9a5e33048"}
Oct 30 01:07:10.668101 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:10.667958    6099 generic.go:356] "Generic (PLEG): container finished" podID="3064087b-a922-4f17-910e-697c6960c3e4" containerID="5c8a879c5d4c5d680d6e96bb067be892b122700709aae19f0b802f186a80b55d" exitCode=0
Oct 30 01:07:10.668101 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:10.668026    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h" event={"ID":"3064087b-a922-4f17-910e-697c6960c3e4","Type":"ContainerDied","Data":"5c8a879c5d4c5d680d6e96bb067be892b122700709aae19f0b802f186a80b55d"}
Oct 30 01:07:10.769171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:10.769141    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn"
Oct 30 01:07:10.842655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:10.842625    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h"
Oct 30 01:07:10.865634 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:10.865607    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c2f8c2-32f1-4acb-a67a-44b18bf39464-kube-api-access-x22qc\" (UniqueName: \"kubernetes.io/projected/c8c2f8c2-32f1-4acb-a67a-44b18bf39464-kube-api-access-x22qc\") pod \"c8c2f8c2-32f1-4acb-a67a-44b18bf39464\" (UID: \"c8c2f8c2-32f1-4acb-a67a-44b18bf39464\") "
Oct 30 01:07:10.865787 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:10.865766    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8c2f8c2-32f1-4acb-a67a-44b18bf39464-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c2f8c2-32f1-4acb-a67a-44b18bf39464-plugins-config-volume\") pod \"c8c2f8c2-32f1-4acb-a67a-44b18bf39464\" (UID: \"c8c2f8c2-32f1-4acb-a67a-44b18bf39464\") "
Oct 30 01:07:10.866137 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:10.866108    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c2f8c2-32f1-4acb-a67a-44b18bf39464-plugins-config-volume" pod "c8c2f8c2-32f1-4acb-a67a-44b18bf39464" (UID: "c8c2f8c2-32f1-4acb-a67a-44b18bf39464"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:10.866620 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:10.866600    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c2f8c2-32f1-4acb-a67a-44b18bf39464-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:10.868065 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:10.868042    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c2f8c2-32f1-4acb-a67a-44b18bf39464-kube-api-access-x22qc" pod "c8c2f8c2-32f1-4acb-a67a-44b18bf39464" (UID: "c8c2f8c2-32f1-4acb-a67a-44b18bf39464"). InnerVolumeSpecName "kube-api-access-x22qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:10.967206 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:10.967170    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3064087b-a922-4f17-910e-697c6960c3e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3064087b-a922-4f17-910e-697c6960c3e4-plugins-config-volume\") pod \"3064087b-a922-4f17-910e-697c6960c3e4\" (UID: \"3064087b-a922-4f17-910e-697c6960c3e4\") "
Oct 30 01:07:10.967399 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:10.967383    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3064087b-a922-4f17-910e-697c6960c3e4-kube-api-access-v45tw\" (UniqueName: \"kubernetes.io/projected/3064087b-a922-4f17-910e-697c6960c3e4-kube-api-access-v45tw\") pod \"3064087b-a922-4f17-910e-697c6960c3e4\" (UID: \"3064087b-a922-4f17-910e-697c6960c3e4\") "
Oct 30 01:07:10.967609 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:10.967581    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3064087b-a922-4f17-910e-697c6960c3e4-plugins-config-volume" pod "3064087b-a922-4f17-910e-697c6960c3e4" (UID: "3064087b-a922-4f17-910e-697c6960c3e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:10.968037 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:10.968018    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x22qc\" (UniqueName: \"kubernetes.io/projected/c8c2f8c2-32f1-4acb-a67a-44b18bf39464-kube-api-access-x22qc\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:10.968107 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:10.968040    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3064087b-a922-4f17-910e-697c6960c3e4-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:10.969418 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:10.969398    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3064087b-a922-4f17-910e-697c6960c3e4-kube-api-access-v45tw" pod "3064087b-a922-4f17-910e-697c6960c3e4" (UID: "3064087b-a922-4f17-910e-697c6960c3e4"). InnerVolumeSpecName "kube-api-access-v45tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:11.069715 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.069629    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v45tw\" (UniqueName: \"kubernetes.io/projected/3064087b-a922-4f17-910e-697c6960c3e4-kube-api-access-v45tw\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:11.130786 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.130754    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"]
Oct 30 01:07:11.131792 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.131748    6099 cpu_manager.go:414] "containerMap: removing container" podUID="c8c2f8c2-32f1-4acb-a67a-44b18bf39464" containerName="payload-processing"
Oct 30 01:07:11.131792 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.131774    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c2f8c2-32f1-4acb-a67a-44b18bf39464" containerName="payload-processing"
Oct 30 01:07:11.132004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.131807    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e351b3f4-02ba-48f9-8298-77e6aa2d148e" containerName="istio-proxy"
Oct 30 01:07:11.132004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.131815    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e351b3f4-02ba-48f9-8298-77e6aa2d148e" containerName="istio-proxy"
Oct 30 01:07:11.132004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.131864    6099 cpu_manager.go:414] "containerMap: removing container" podUID="744aef66-2b84-4c2b-951e-97d85260e902" containerName="istio-proxy"
Oct 30 01:07:11.132004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.131871    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="744aef66-2b84-4c2b-951e-97d85260e902" containerName="istio-proxy"
Oct 30 01:07:11.132004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.131902    6099 cpu_manager.go:414] "containerMap: removing container" podUID="49f719fe-16aa-431d-91b7-1730267521ec" containerName="maas-api"
Oct 30 01:07:11.132004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.131908    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f719fe-16aa-431d-91b7-1730267521ec" containerName="maas-api"
Oct 30 01:07:11.132004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.131926    6099 cpu_manager.go:414] "containerMap: removing container" podUID="3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83" containerName="maas-api"
Oct 30 01:07:11.132004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.131933    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83" containerName="maas-api"
Oct 30 01:07:11.132004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.131945    6099 cpu_manager.go:414] "containerMap: removing container" podUID="04f57545-b25c-45ac-9fd5-bc9c6e1cf068" containerName="istio-proxy"
Oct 30 01:07:11.132004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.131951    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f57545-b25c-45ac-9fd5-bc9c6e1cf068" containerName="istio-proxy"
Oct 30 01:07:11.132004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.131977    6099 cpu_manager.go:414] "containerMap: removing container" podUID="394e8301-d15d-46d4-aa5c-10f3e154c948" containerName="istio-proxy"
Oct 30 01:07:11.132004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.131984    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394e8301-d15d-46d4-aa5c-10f3e154c948" containerName="istio-proxy"
Oct 30 01:07:11.132004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.132006    6099 cpu_manager.go:414] "containerMap: removing container" podUID="3064087b-a922-4f17-910e-697c6960c3e4" containerName="payload-pre-processing"
Oct 30 01:07:11.132632 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.132014    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3064087b-a922-4f17-910e-697c6960c3e4" containerName="payload-pre-processing"
Oct 30 01:07:11.132632 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.132207    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e351b3f4-02ba-48f9-8298-77e6aa2d148e" containerName="istio-proxy"
Oct 30 01:07:11.132632 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.132228    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="744aef66-2b84-4c2b-951e-97d85260e902" containerName="istio-proxy"
Oct 30 01:07:11.132632 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.132248    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f719fe-16aa-431d-91b7-1730267521ec" containerName="maas-api"
Oct 30 01:07:11.132632 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.132261    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b2a1b79-8e4f-4b19-9cc3-8c34d48d7e83" containerName="maas-api"
Oct 30 01:07:11.132632 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.132270    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f57545-b25c-45ac-9fd5-bc9c6e1cf068" containerName="istio-proxy"
Oct 30 01:07:11.132632 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.132284    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="394e8301-d15d-46d4-aa5c-10f3e154c948" containerName="istio-proxy"
Oct 30 01:07:11.132632 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.132297    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3064087b-a922-4f17-910e-697c6960c3e4" containerName="payload-pre-processing"
Oct 30 01:07:11.132632 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.132310    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c2f8c2-32f1-4acb-a67a-44b18bf39464" containerName="payload-processing"
Oct 30 01:07:11.136522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.136502    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.163015 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.162983    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f96ede-openshift-default-dockercfg-47dgx\""
Oct 30 01:07:11.169207 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.169182    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"]
Oct 30 01:07:11.272861 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.272825    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-credential-socket\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.273038 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.272877    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-istio-envoy\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.273038 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.272953    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-istio-data\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.273116 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.273037    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edf31a9e-34ff-428b-9c36-4189119697d7-istiod-ca-cert\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.273116 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.273068    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd2l5\" (UniqueName: \"kubernetes.io/projected/edf31a9e-34ff-428b-9c36-4189119697d7-kube-api-access-pd2l5\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.273186 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.273128    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edf31a9e-34ff-428b-9c36-4189119697d7-istio-token\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.273186 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.273169    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-workload-certs\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.273255 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.273246    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-workload-socket\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.273325 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.273314    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edf31a9e-34ff-428b-9c36-4189119697d7-istio-podinfo\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.375665 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.375557    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-credential-socket\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.375665 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.375615    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-istio-envoy\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.375665 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.375647    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-istio-data\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.376040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.375728    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edf31a9e-34ff-428b-9c36-4189119697d7-istiod-ca-cert\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.376040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.375765    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd2l5\" (UniqueName: \"kubernetes.io/projected/edf31a9e-34ff-428b-9c36-4189119697d7-kube-api-access-pd2l5\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.376040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.375820    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edf31a9e-34ff-428b-9c36-4189119697d7-istio-token\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.376040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.375881    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-workload-certs\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.376040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.375923    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-workload-socket\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.376306 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.376034    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edf31a9e-34ff-428b-9c36-4189119697d7-istio-podinfo\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.376857 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.376832    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-istio-data\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.377050 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.376831    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-workload-socket\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.377200 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.376936    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-workload-certs\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.377396 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.377375    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edf31a9e-34ff-428b-9c36-4189119697d7-istiod-ca-cert\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.377488 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.377471    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-credential-socket\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.379550 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.379453    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-istio-envoy\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.380478 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.380453    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edf31a9e-34ff-428b-9c36-4189119697d7-istio-podinfo\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.399125 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.399098    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edf31a9e-34ff-428b-9c36-4189119697d7-istio-token\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.399309 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.399288    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd2l5\" (UniqueName: \"kubernetes.io/projected/edf31a9e-34ff-428b-9c36-4189119697d7-kube-api-access-pd2l5\") pod \"e2e-inf-b-f96ede-openshift-default-684654576c-whs76\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") " pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.450796 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.450767    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:11.678922 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.678840    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn"
Oct 30 01:07:11.678922 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.678865    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn" event={"ID":"c8c2f8c2-32f1-4acb-a67a-44b18bf39464","Type":"ContainerDied","Data":"6060a73843696ace308b9343effe23a739949d1746ac12d30efec7d7f9358ba3"}
Oct 30 01:07:11.679428 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.678943    6099 scope.go:122] "RemoveContainer" containerID="fd2435d0df5c386a00f1c8e11f28bc713086893a10380780385228d9a5e33048"
Oct 30 01:07:11.680639 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.680613    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h"
Oct 30 01:07:11.680639 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.680625    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h" event={"ID":"3064087b-a922-4f17-910e-697c6960c3e4","Type":"ContainerDied","Data":"0d72f6c8be72e2d56f36947e6bdcc419808f13a12f966a8e95125837d2d90abc"}
Oct 30 01:07:11.691789 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.691769    6099 scope.go:122] "RemoveContainer" containerID="5c8a879c5d4c5d680d6e96bb067be892b122700709aae19f0b802f186a80b55d"
Oct 30 01:07:11.730906 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.730873    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h"]
Oct 30 01:07:11.734521 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.734497    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-43c02d-f765c975-f6d2h"]
Oct 30 01:07:11.768805 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.768756    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn"]
Oct 30 01:07:11.772337 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.772309    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-43c02d-7c8c8448c6-4bbfn"]
Oct 30 01:07:11.777845 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.777822    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"]
Oct 30 01:07:11.782917 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:07:11.782893    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf31a9e_34ff_428b_9c36_4189119697d7.slice/crio-988816cce779e3973d8ed31f3270b9765d6cac875fb97a86fcb7ea6314262c7b WatchSource:0}: Error finding container 988816cce779e3973d8ed31f3270b9765d6cac875fb97a86fcb7ea6314262c7b: Status 404 returned error can't find the container with id 988816cce779e3973d8ed31f3270b9765d6cac875fb97a86fcb7ea6314262c7b
Oct 30 01:07:11.786114 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.786081    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:07:11.786193 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.786145    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:07:11.786193 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:11.786176    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:07:12.267095 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:12.267050    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3064087b-a922-4f17-910e-697c6960c3e4" path="/var/lib/kubelet/pods/3064087b-a922-4f17-910e-697c6960c3e4/volumes"
Oct 30 01:07:12.267720 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:12.267683    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8c2f8c2-32f1-4acb-a67a-44b18bf39464" path="/var/lib/kubelet/pods/c8c2f8c2-32f1-4acb-a67a-44b18bf39464/volumes"
Oct 30 01:07:12.534167 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:12.534065    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw"]
Oct 30 01:07:12.593928 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:12.593882    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc"]
Oct 30 01:07:12.703932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:12.703895    6099 scope.go:122] "RemoveContainer" containerID="f02cc3d854d25a2bf0238e62ab2ec4717de3ee72cc510fc8d7349735d8273c09"
Oct 30 01:07:12.718503 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:12.718451    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76" event={"ID":"edf31a9e-34ff-428b-9c36-4189119697d7","Type":"ContainerStarted","Data":"acd4064f9bfa8deb15f1561659227bddcf7d83b36d3eb61f22ed9cdcf9259449"}
Oct 30 01:07:12.719017 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:12.718974    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76" event={"ID":"edf31a9e-34ff-428b-9c36-4189119697d7","Type":"ContainerStarted","Data":"988816cce779e3973d8ed31f3270b9765d6cac875fb97a86fcb7ea6314262c7b"}
Oct 30 01:07:12.785782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:12.780367    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76" podStartSLOduration=1.780346741 podStartE2EDuration="1.780346741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:12.778101118 +0000 UTC m=+1926.635008874" watchObservedRunningTime="2026-10-30 01:07:12.780346741 +0000 UTC m=+1926.637254494"
Oct 30 01:07:13.460731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:13.452155    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:13.460731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:13.459764    6099 patch_prober.go:29] interesting pod/e2e-inf-b-f96ede-openshift-default-684654576c-whs76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Oct 30 01:07:13.460731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:13.459822    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76" podUID="edf31a9e-34ff-428b-9c36-4189119697d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Oct 30 01:07:14.455739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:14.454883    6099 patch_prober.go:29] interesting pod/e2e-inf-b-f96ede-openshift-default-684654576c-whs76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Oct 30 01:07:14.455739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:14.454954    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76" podUID="edf31a9e-34ff-428b-9c36-4189119697d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Oct 30 01:07:15.295742 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.295688    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc"]
Oct 30 01:07:15.302883 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.302859    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc"
Oct 30 01:07:15.311661 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.311637    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc"]
Oct 30 01:07:15.319574 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.319551    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:07:15.319753 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.319608    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:07:15.321916 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.321892    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc67105e-hhbcc\" (UID: \"83eafb1e-2ca8-4a11-bea0-04ed87a35dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc"
Oct 30 01:07:15.322125 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.322096    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc67105e-hhbcc\" (UID: \"83eafb1e-2ca8-4a11-bea0-04ed87a35dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc"
Oct 30 01:07:15.322229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.322210    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkngg\" (UniqueName: \"kubernetes.io/projected/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-kube-api-access-lkngg\") pod \"maas-api-revoke-sub-fc67105e-hhbcc\" (UID: \"83eafb1e-2ca8-4a11-bea0-04ed87a35dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc"
Oct 30 01:07:15.424099 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.424067    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc67105e-hhbcc\" (UID: \"83eafb1e-2ca8-4a11-bea0-04ed87a35dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc"
Oct 30 01:07:15.424327 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.424313    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc67105e-hhbcc\" (UID: \"83eafb1e-2ca8-4a11-bea0-04ed87a35dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc"
Oct 30 01:07:15.424426 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.424416    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkngg\" (UniqueName: \"kubernetes.io/projected/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-kube-api-access-lkngg\") pod \"maas-api-revoke-sub-fc67105e-hhbcc\" (UID: \"83eafb1e-2ca8-4a11-bea0-04ed87a35dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc"
Oct 30 01:07:15.425004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.424982    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc67105e-hhbcc\" (UID: \"83eafb1e-2ca8-4a11-bea0-04ed87a35dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc"
Oct 30 01:07:15.426401 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.426380    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc67105e-hhbcc\" (UID: \"83eafb1e-2ca8-4a11-bea0-04ed87a35dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc"
Oct 30 01:07:15.449455 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.449433    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkngg\" (UniqueName: \"kubernetes.io/projected/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-kube-api-access-lkngg\") pod \"maas-api-revoke-sub-fc67105e-hhbcc\" (UID: \"83eafb1e-2ca8-4a11-bea0-04ed87a35dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc"
Oct 30 01:07:15.451363 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.451337    6099 patch_prober.go:29] interesting pod/e2e-inf-b-f96ede-openshift-default-684654576c-whs76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Oct 30 01:07:15.451439 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.451377    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76" podUID="edf31a9e-34ff-428b-9c36-4189119697d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Oct 30 01:07:15.622269 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.622010    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc"
Oct 30 01:07:15.943242 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.943165    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"]
Oct 30 01:07:15.949315 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.949294    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:15.987733 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:15.982050    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-969b92-openshift-default-dockercfg-4fk78\""
Oct 30 01:07:15.996183 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:07:15.996154    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83eafb1e_2ca8_4a11_bea0_04ed87a35dab.slice/crio-d8ee512c40876fd0973354d6df5b01dde58af62f0e3d42781edfd8454a44c850 WatchSource:0}: Error finding container d8ee512c40876fd0973354d6df5b01dde58af62f0e3d42781edfd8454a44c850: Status 404 returned error can't find the container with id d8ee512c40876fd0973354d6df5b01dde58af62f0e3d42781edfd8454a44c850
Oct 30 01:07:16.000690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.000660    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc"]
Oct 30 01:07:16.035582 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.035551    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-istio-data\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.035735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.035596    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqjzr\" (UniqueName: \"kubernetes.io/projected/ed244c34-4856-46d8-a180-030fd7d4362e-kube-api-access-jqjzr\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.035735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.035658    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-workload-certs\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.035735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.035723    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-workload-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.035917 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.035850    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed244c34-4856-46d8-a180-030fd7d4362e-istiod-ca-cert\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.035974 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.035938    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-credential-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.036035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.036022    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-istio-envoy\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.036091 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.036046    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed244c34-4856-46d8-a180-030fd7d4362e-istio-podinfo\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.036176 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.036158    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed244c34-4856-46d8-a180-030fd7d4362e-istio-token\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.036766 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.036741    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"]
Oct 30 01:07:16.139127 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.139091    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-istio-data\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.139351 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.139335    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqjzr\" (UniqueName: \"kubernetes.io/projected/ed244c34-4856-46d8-a180-030fd7d4362e-kube-api-access-jqjzr\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.139486 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.139472    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-workload-certs\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.139630 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.139606    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-workload-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.139838 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.139490    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-istio-data\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.139935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.139916    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-workload-certs\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.140005 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.139973    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-workload-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.140057 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.140036    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed244c34-4856-46d8-a180-030fd7d4362e-istiod-ca-cert\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.140109 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.140086    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-credential-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.140148 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.140132    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-istio-envoy\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.140193 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.140149    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed244c34-4856-46d8-a180-030fd7d4362e-istio-podinfo\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.140231 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.140212    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed244c34-4856-46d8-a180-030fd7d4362e-istio-token\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.140609 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.140588    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-credential-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.140949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.140929    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed244c34-4856-46d8-a180-030fd7d4362e-istiod-ca-cert\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.142719 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.142688    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-istio-envoy\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.143068 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.143046    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed244c34-4856-46d8-a180-030fd7d4362e-istio-podinfo\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.162492 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.162465    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqjzr\" (UniqueName: \"kubernetes.io/projected/ed244c34-4856-46d8-a180-030fd7d4362e-kube-api-access-jqjzr\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.163186 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.163163    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed244c34-4856-46d8-a180-030fd7d4362e-istio-token\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.284104 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.284023    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:16.458990 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.458954    6099 patch_prober.go:29] interesting pod/e2e-inf-b-f96ede-openshift-default-684654576c-whs76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Oct 30 01:07:16.459245 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.459215    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76" podUID="edf31a9e-34ff-428b-9c36-4189119697d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Oct 30 01:07:16.624519 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.624488    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"]
Oct 30 01:07:16.630266 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.630221    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:07:16.630393 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.630316    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:07:16.630393 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.630364    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:07:16.636701 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:07:16.636670    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:07:16.761193 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.761117    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc" event={"ID":"83eafb1e-2ca8-4a11-bea0-04ed87a35dab","Type":"ContainerStarted","Data":"8e6154796d5712c38171030455b6cccf6e503ffd38e689cec46873a484059df1"}
Oct 30 01:07:16.761462 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.761418    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc" event={"ID":"83eafb1e-2ca8-4a11-bea0-04ed87a35dab","Type":"ContainerStarted","Data":"d8ee512c40876fd0973354d6df5b01dde58af62f0e3d42781edfd8454a44c850"}
Oct 30 01:07:16.763660 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.763634    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x" event={"ID":"ed244c34-4856-46d8-a180-030fd7d4362e","Type":"ContainerStarted","Data":"fd049efca35e876e1ce093c359eeb2f77c28ba75ab448e497f9b96c9b7dcc672"}
Oct 30 01:07:16.763802 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.763676    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x" event={"ID":"ed244c34-4856-46d8-a180-030fd7d4362e","Type":"ContainerStarted","Data":"eef73b7a14183d74906efbb7166c5ed1ab01bb2fa8ff7f012172507f5d9a312d"}
Oct 30 01:07:16.813721 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.812779    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc" podStartSLOduration=1.812760329 podStartE2EDuration="1.812760329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:16.811228886 +0000 UTC m=+1930.668136641" watchObservedRunningTime="2026-10-30 01:07:16.812760329 +0000 UTC m=+1930.669668086"
Oct 30 01:07:16.892949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.887347    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"]
Oct 30 01:07:16.907730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.904961    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:16.922156 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.917980    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x" podStartSLOduration=1.917956797 podStartE2EDuration="1.917956797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:16.863242171 +0000 UTC m=+1930.720149927" watchObservedRunningTime="2026-10-30 01:07:16.917956797 +0000 UTC m=+1930.774864555"
Oct 30 01:07:16.940924 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.923283    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1da7b2-openshift-default-dockercfg-zwj8q\""
Oct 30 01:07:16.940924 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.935700    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"]
Oct 30 01:07:16.956132 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.956097    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-workload-socket\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:16.956320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.956263    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-podinfo\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:16.956320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.956299    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-credential-socket\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:16.956440 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.956352    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-workload-certs\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:16.956440 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.956371    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7451c167-4bdb-44dd-96ab-ed72782229d0-istiod-ca-cert\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:16.956440 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.956426    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-envoy\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:16.956689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.956471    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-token\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:16.956689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.956518    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-data\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:16.956689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:16.956542    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk7vl\" (UniqueName: \"kubernetes.io/projected/7451c167-4bdb-44dd-96ab-ed72782229d0-kube-api-access-sk7vl\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.059756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.057936    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-podinfo\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.059756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.058009    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-credential-socket\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.059756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.058077    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-workload-certs\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.059756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.058103    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7451c167-4bdb-44dd-96ab-ed72782229d0-istiod-ca-cert\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.059756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.058191    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-envoy\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.059756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.058259    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-token\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.059756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.058333    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-data\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.059756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.058369    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk7vl\" (UniqueName: \"kubernetes.io/projected/7451c167-4bdb-44dd-96ab-ed72782229d0-kube-api-access-sk7vl\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.059756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.059418    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7451c167-4bdb-44dd-96ab-ed72782229d0-istiod-ca-cert\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.061120 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.060135    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-data\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.061120 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.060667    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-credential-socket\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.089056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.084036    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-workload-socket\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.089056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.084354    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-workload-certs\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.108517 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.108445    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-envoy\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.108854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.108821    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-workload-socket\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.111309 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.111226    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk7vl\" (UniqueName: \"kubernetes.io/projected/7451c167-4bdb-44dd-96ab-ed72782229d0-kube-api-access-sk7vl\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.115315 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.115281    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-podinfo\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.119903 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.119884    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"]
Oct 30 01:07:17.120386 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.120205    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t" podUID="fbad3465-dc78-4586-a4dc-071709ad50dd" containerName="istio-proxy" containerID="cri-o://e52e84720efbc78c384abcb7bdcde2ff362e292f8080e637fdab1d64289a55c5" gracePeriod=30
Oct 30 01:07:17.133157 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.133132    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-token\") pod \"e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") " pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.284779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.282175    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:17.310783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.300099    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:17.320727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.317961    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 30 01:07:17.320727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.318020    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x" podUID="ed244c34-4856-46d8-a180-030fd7d4362e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 30 01:07:17.484728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.480944    6099 patch_prober.go:29] interesting pod/e2e-inf-b-f96ede-openshift-default-684654576c-whs76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Oct 30 01:07:17.484728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.481011    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76" podUID="edf31a9e-34ff-428b-9c36-4189119697d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Oct 30 01:07:17.562010 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.560849    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw" podUID="17244f1e-2983-4624-83b3-955995424494" containerName="payload-processing" containerID="cri-o://7fbc0f38c8852e8cd7551b9baa7ea0b9370def39822d93a4a98b19772ab14bfe" gracePeriod=25
Oct 30 01:07:17.594740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.594330    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc" podUID="ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f" containerName="payload-pre-processing" containerID="cri-o://7984101ec73e2d7e256b925fde5ad2f0dfbae19f94cec37e48e2c445e84c9328" gracePeriod=25
Oct 30 01:07:17.939730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.932151    6099 generic.go:356] "Generic (PLEG): container finished" podID="83eafb1e-2ca8-4a11-bea0-04ed87a35dab" containerID="8e6154796d5712c38171030455b6cccf6e503ffd38e689cec46873a484059df1" exitCode=0
Oct 30 01:07:17.939730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.932320    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc" event={"ID":"83eafb1e-2ca8-4a11-bea0-04ed87a35dab","Type":"ContainerDied","Data":"8e6154796d5712c38171030455b6cccf6e503ffd38e689cec46873a484059df1"}
Oct 30 01:07:17.939730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.934919    6099 generic.go:356] "Generic (PLEG): container finished" podID="17244f1e-2983-4624-83b3-955995424494" containerID="7fbc0f38c8852e8cd7551b9baa7ea0b9370def39822d93a4a98b19772ab14bfe" exitCode=0
Oct 30 01:07:17.939730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:17.935762    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw" event={"ID":"17244f1e-2983-4624-83b3-955995424494","Type":"ContainerDied","Data":"7fbc0f38c8852e8cd7551b9baa7ea0b9370def39822d93a4a98b19772ab14bfe"}
Oct 30 01:07:18.078820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.070817    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"]
Oct 30 01:07:18.086969 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.086927    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:07:18.088800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.088769    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:07:18.089008 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.088984    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:07:18.227485 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.227456    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw"
Oct 30 01:07:18.255729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.253982    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17244f1e-2983-4624-83b3-955995424494-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17244f1e-2983-4624-83b3-955995424494-plugins-config-volume\") pod \"17244f1e-2983-4624-83b3-955995424494\" (UID: \"17244f1e-2983-4624-83b3-955995424494\") "
Oct 30 01:07:18.255729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.254148    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17244f1e-2983-4624-83b3-955995424494-kube-api-access-zk8qk\" (UniqueName: \"kubernetes.io/projected/17244f1e-2983-4624-83b3-955995424494-kube-api-access-zk8qk\") pod \"17244f1e-2983-4624-83b3-955995424494\" (UID: \"17244f1e-2983-4624-83b3-955995424494\") "
Oct 30 01:07:18.255729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.255401    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17244f1e-2983-4624-83b3-955995424494-plugins-config-volume" pod "17244f1e-2983-4624-83b3-955995424494" (UID: "17244f1e-2983-4624-83b3-955995424494"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:18.257890 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.257180    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17244f1e-2983-4624-83b3-955995424494-kube-api-access-zk8qk" pod "17244f1e-2983-4624-83b3-955995424494" (UID: "17244f1e-2983-4624-83b3-955995424494"). InnerVolumeSpecName "kube-api-access-zk8qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:18.291597 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.290192    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 30 01:07:18.291597 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.290253    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x" podUID="ed244c34-4856-46d8-a180-030fd7d4362e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 30 01:07:18.356320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.356288    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17244f1e-2983-4624-83b3-955995424494-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:18.356553 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.356526    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk8qk\" (UniqueName: \"kubernetes.io/projected/17244f1e-2983-4624-83b3-955995424494-kube-api-access-zk8qk\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:18.452017 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.451982    6099 patch_prober.go:29] interesting pod/e2e-inf-b-f96ede-openshift-default-684654576c-whs76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Oct 30 01:07:18.452174 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.452045    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76" podUID="edf31a9e-34ff-428b-9c36-4189119697d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Oct 30 01:07:18.946901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.946853    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw"
Oct 30 01:07:18.946901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.946887    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw" event={"ID":"17244f1e-2983-4624-83b3-955995424494","Type":"ContainerDied","Data":"354ee0e6c67da25eed5757f7d9afc933c939ff9b67a35d363d16317a1672a271"}
Oct 30 01:07:18.947342 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.946954    6099 scope.go:122] "RemoveContainer" containerID="7fbc0f38c8852e8cd7551b9baa7ea0b9370def39822d93a4a98b19772ab14bfe"
Oct 30 01:07:18.948777 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.948752    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8" event={"ID":"7451c167-4bdb-44dd-96ab-ed72782229d0","Type":"ContainerStarted","Data":"c01b4f1bf335b3e4bcba294183c93c533fd0fb245fe7c0c52e3e29c843a245be"}
Oct 30 01:07:18.948853 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:18.948804    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8" event={"ID":"7451c167-4bdb-44dd-96ab-ed72782229d0","Type":"ContainerStarted","Data":"8df930588df596700c6ed5446e74d9cf1b9625a8e6019f008e1fca6e9b17b9e2"}
Oct 30 01:07:19.050559 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.050508    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8" podStartSLOduration=3.050495326 podStartE2EDuration="3.050495326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:19.04963567 +0000 UTC m=+1932.906543424" watchObservedRunningTime="2026-10-30 01:07:19.050495326 +0000 UTC m=+1932.907403080"
Oct 30 01:07:19.133418 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.132867    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw"]
Oct 30 01:07:19.138472 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.138419    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a07920-5657fc66dd-vk2lw"]
Oct 30 01:07:19.275789 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.275134    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc"
Oct 30 01:07:19.304744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.302656    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:19.304744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.304052    6099 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 30 01:07:19.304744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.304121    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8" podUID="7451c167-4bdb-44dd-96ab-ed72782229d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 30 01:07:19.304744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.304429    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 30 01:07:19.304744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.304461    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x" podUID="ed244c34-4856-46d8-a180-030fd7d4362e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 30 01:07:19.434841 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.434810    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-kube-api-access-lkngg\" (UniqueName: \"kubernetes.io/projected/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-kube-api-access-lkngg\") pod \"83eafb1e-2ca8-4a11-bea0-04ed87a35dab\" (UID: \"83eafb1e-2ca8-4a11-bea0-04ed87a35dab\") "
Oct 30 01:07:19.435274 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.435246    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-maas-api-xks-ca\") pod \"83eafb1e-2ca8-4a11-bea0-04ed87a35dab\" (UID: \"83eafb1e-2ca8-4a11-bea0-04ed87a35dab\") "
Oct 30 01:07:19.435427 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.435315    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-maas-api-openshift-ca\") pod \"83eafb1e-2ca8-4a11-bea0-04ed87a35dab\" (UID: \"83eafb1e-2ca8-4a11-bea0-04ed87a35dab\") "
Oct 30 01:07:19.435889 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.435861    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-maas-api-openshift-ca" pod "83eafb1e-2ca8-4a11-bea0-04ed87a35dab" (UID: "83eafb1e-2ca8-4a11-bea0-04ed87a35dab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:19.436218 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.436195    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:19.437772 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.437749    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-maas-api-xks-ca" pod "83eafb1e-2ca8-4a11-bea0-04ed87a35dab" (UID: "83eafb1e-2ca8-4a11-bea0-04ed87a35dab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:07:19.438442 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.438417    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-kube-api-access-lkngg" pod "83eafb1e-2ca8-4a11-bea0-04ed87a35dab" (UID: "83eafb1e-2ca8-4a11-bea0-04ed87a35dab"). InnerVolumeSpecName "kube-api-access-lkngg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:19.458555 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.458523    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:19.459940 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.459920    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:19.460683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.460665    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:07:19.537603 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.537528    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:19.537603 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.537554    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkngg\" (UniqueName: \"kubernetes.io/projected/83eafb1e-2ca8-4a11-bea0-04ed87a35dab-kube-api-access-lkngg\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:19.958318 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.958283    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc" event={"ID":"83eafb1e-2ca8-4a11-bea0-04ed87a35dab","Type":"ContainerDied","Data":"d8ee512c40876fd0973354d6df5b01dde58af62f0e3d42781edfd8454a44c850"}
Oct 30 01:07:19.958727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.958337    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8ee512c40876fd0973354d6df5b01dde58af62f0e3d42781edfd8454a44c850"
Oct 30 01:07:19.958727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:19.958302    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc"
Oct 30 01:07:20.287780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.265035    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17244f1e-2983-4624-83b3-955995424494" path="/var/lib/kubelet/pods/17244f1e-2983-4624-83b3-955995424494/volumes"
Oct 30 01:07:20.287780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.286999    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 30 01:07:20.287780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.287051    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x" podUID="ed244c34-4856-46d8-a180-030fd7d4362e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 30 01:07:20.312731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.312157    6099 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 30 01:07:20.312731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.312212    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8" podUID="7451c167-4bdb-44dd-96ab-ed72782229d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 30 01:07:20.422157 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.420825    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"]
Oct 30 01:07:20.422157 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.422115    6099 cpu_manager.go:414] "containerMap: removing container" podUID="83eafb1e-2ca8-4a11-bea0-04ed87a35dab" containerName="revoke-keys"
Oct 30 01:07:20.422157 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.422139    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83eafb1e-2ca8-4a11-bea0-04ed87a35dab" containerName="revoke-keys"
Oct 30 01:07:20.422482 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.422197    6099 cpu_manager.go:414] "containerMap: removing container" podUID="17244f1e-2983-4624-83b3-955995424494" containerName="payload-processing"
Oct 30 01:07:20.422482 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.422206    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17244f1e-2983-4624-83b3-955995424494" containerName="payload-processing"
Oct 30 01:07:20.422482 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.422432    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="83eafb1e-2ca8-4a11-bea0-04ed87a35dab" containerName="revoke-keys"
Oct 30 01:07:20.422482 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.422457    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="17244f1e-2983-4624-83b3-955995424494" containerName="payload-processing"
Oct 30 01:07:20.430009 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.429984    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"]
Oct 30 01:07:20.430173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.430096    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.447016 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.444884    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-49dfaf-openshift-default-dockercfg-5v6p4\""
Oct 30 01:07:20.561646 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.561611    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-workload-socket\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.561989 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.561971    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-credential-socket\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.562098 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.562088    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f66406e6-2198-443d-8130-31b590e4f566-istio-token\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.562202 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.562191    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f66406e6-2198-443d-8130-31b590e4f566-istiod-ca-cert\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.562349 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.562323    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-workload-certs\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.562439 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.562427    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-istio-envoy\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.562565 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.562544    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f66406e6-2198-443d-8130-31b590e4f566-istio-podinfo\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.562604 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.562586    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnwsh\" (UniqueName: \"kubernetes.io/projected/f66406e6-2198-443d-8130-31b590e4f566-kube-api-access-jnwsh\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.562649 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.562637    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-istio-data\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.663685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.663641    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-credential-socket\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.663932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.663698    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f66406e6-2198-443d-8130-31b590e4f566-istio-token\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.663932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.663757    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f66406e6-2198-443d-8130-31b590e4f566-istiod-ca-cert\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.663932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.663812    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-workload-certs\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.663932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.663914    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-istio-envoy\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.663932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.663931    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f66406e6-2198-443d-8130-31b590e4f566-istio-podinfo\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.664218 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.663960    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnwsh\" (UniqueName: \"kubernetes.io/projected/f66406e6-2198-443d-8130-31b590e4f566-kube-api-access-jnwsh\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.664218 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.663988    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-istio-data\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.664218 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.664031    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-workload-socket\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.664218 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.664037    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-credential-socket\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.664432 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.664273    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-workload-socket\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.688739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.664900    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-istio-data\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.688739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.664962    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-workload-certs\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.688739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.665336    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f66406e6-2198-443d-8130-31b590e4f566-istiod-ca-cert\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.723836 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.723810    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f66406e6-2198-443d-8130-31b590e4f566-istio-podinfo\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.725182 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.725145    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-istio-envoy\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.725745 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.725679    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnwsh\" (UniqueName: \"kubernetes.io/projected/f66406e6-2198-443d-8130-31b590e4f566-kube-api-access-jnwsh\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.726034 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.726017    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f66406e6-2198-443d-8130-31b590e4f566-istio-token\") pod \"e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") " pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:20.750469 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:20.750445    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:21.219138 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:21.219028    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"]
Oct 30 01:07:21.223210 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:07:21.223183    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66406e6_2198_443d_8130_31b590e4f566.slice/crio-384f6b667312ee1e5225ab3fd503a2562cd30ffd9a34bf9c0f72a97d0437c0f2 WatchSource:0}: Error finding container 384f6b667312ee1e5225ab3fd503a2562cd30ffd9a34bf9c0f72a97d0437c0f2: Status 404 returned error can't find the container with id 384f6b667312ee1e5225ab3fd503a2562cd30ffd9a34bf9c0f72a97d0437c0f2
Oct 30 01:07:21.226673 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:21.226636    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:07:21.226796 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:21.226735    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:07:21.226796 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:21.226788    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:07:21.300934 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:21.286843    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 30 01:07:21.300934 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:21.286905    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x" podUID="ed244c34-4856-46d8-a180-030fd7d4362e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 30 01:07:21.324507 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:21.324167    6099 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 30 01:07:21.324507 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:21.324238    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8" podUID="7451c167-4bdb-44dd-96ab-ed72782229d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 30 01:07:22.203666 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:22.203625    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj" event={"ID":"f66406e6-2198-443d-8130-31b590e4f566","Type":"ContainerStarted","Data":"c81ee7acbcbadc3ac360ad5f91973475a9313d8d30ba27bc037910e69b8a692c"}
Oct 30 01:07:22.211363 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:22.211332    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj" event={"ID":"f66406e6-2198-443d-8130-31b590e4f566","Type":"ContainerStarted","Data":"384f6b667312ee1e5225ab3fd503a2562cd30ffd9a34bf9c0f72a97d0437c0f2"}
Oct 30 01:07:22.303732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:22.303094    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj" podStartSLOduration=2.303073099 podStartE2EDuration="2.303073099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:22.277674782 +0000 UTC m=+1936.134582538" watchObservedRunningTime="2026-10-30 01:07:22.303073099 +0000 UTC m=+1936.159980855"
Oct 30 01:07:22.320751 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:22.320215    6099 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 30 01:07:22.320751 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:22.320280    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8" podUID="7451c167-4bdb-44dd-96ab-ed72782229d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 30 01:07:22.346099 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:22.344855    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 30 01:07:22.346099 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:22.344910    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x" podUID="ed244c34-4856-46d8-a180-030fd7d4362e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 30 01:07:22.813049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:22.791734    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:22.813049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:22.802048    6099 patch_prober.go:29] interesting pod/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Oct 30 01:07:22.813049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:22.802109    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj" podUID="f66406e6-2198-443d-8130-31b590e4f566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Oct 30 01:07:23.158799 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.158356    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc" podUID="ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:07:23.309731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.308655    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 30 01:07:23.309731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.309488    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x" podUID="ed244c34-4856-46d8-a180-030fd7d4362e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 30 01:07:23.358324 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.358288    6099 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 30 01:07:23.358542 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.358346    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8" podUID="7451c167-4bdb-44dd-96ab-ed72782229d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 30 01:07:23.364039 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.363997    6099 generic.go:356] "Generic (PLEG): container finished" podID="fbad3465-dc78-4586-a4dc-071709ad50dd" containerID="e52e84720efbc78c384abcb7bdcde2ff362e292f8080e637fdab1d64289a55c5" exitCode=0
Oct 30 01:07:23.364611 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.364556    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:07:23.364611 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.364576    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t" event={"ID":"fbad3465-dc78-4586-a4dc-071709ad50dd","Type":"ContainerDied","Data":"e52e84720efbc78c384abcb7bdcde2ff362e292f8080e637fdab1d64289a55c5"}
Oct 30 01:07:23.364781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.364642    6099 scope.go:122] "RemoveContainer" containerID="e52e84720efbc78c384abcb7bdcde2ff362e292f8080e637fdab1d64289a55c5"
Oct 30 01:07:23.479745 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.467547    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-workload-socket\") pod \"fbad3465-dc78-4586-a4dc-071709ad50dd\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") "
Oct 30 01:07:23.479745 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.467623    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbad3465-dc78-4586-a4dc-071709ad50dd-kube-api-access-rdtf2\" (UniqueName: \"kubernetes.io/projected/fbad3465-dc78-4586-a4dc-071709ad50dd-kube-api-access-rdtf2\") pod \"fbad3465-dc78-4586-a4dc-071709ad50dd\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") "
Oct 30 01:07:23.479745 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.472617    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-workload-socket" pod "fbad3465-dc78-4586-a4dc-071709ad50dd" (UID: "fbad3465-dc78-4586-a4dc-071709ad50dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:23.506209 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.499447    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-envoy\") pod \"fbad3465-dc78-4586-a4dc-071709ad50dd\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") "
Oct 30 01:07:23.506209 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.499656    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-credential-socket\") pod \"fbad3465-dc78-4586-a4dc-071709ad50dd\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") "
Oct 30 01:07:23.506209 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.499801    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-workload-certs\") pod \"fbad3465-dc78-4586-a4dc-071709ad50dd\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") "
Oct 30 01:07:23.506209 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.499844    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-token\" (UniqueName: \"kubernetes.io/projected/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-token\") pod \"fbad3465-dc78-4586-a4dc-071709ad50dd\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") "
Oct 30 01:07:23.515857 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.515819    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-workload-certs" pod "fbad3465-dc78-4586-a4dc-071709ad50dd" (UID: "fbad3465-dc78-4586-a4dc-071709ad50dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:23.518931 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.518901    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-credential-socket" pod "fbad3465-dc78-4586-a4dc-071709ad50dd" (UID: "fbad3465-dc78-4586-a4dc-071709ad50dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:23.519184 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.519168    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbad3465-dc78-4586-a4dc-071709ad50dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbad3465-dc78-4586-a4dc-071709ad50dd-istiod-ca-cert\") pod \"fbad3465-dc78-4586-a4dc-071709ad50dd\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") "
Oct 30 01:07:23.519669 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.519653    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-data\") pod \"fbad3465-dc78-4586-a4dc-071709ad50dd\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") "
Oct 30 01:07:23.519862 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.519846    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-podinfo\") pod \"fbad3465-dc78-4586-a4dc-071709ad50dd\" (UID: \"fbad3465-dc78-4586-a4dc-071709ad50dd\") "
Oct 30 01:07:23.536454 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.519603    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbad3465-dc78-4586-a4dc-071709ad50dd-istiod-ca-cert" pod "fbad3465-dc78-4586-a4dc-071709ad50dd" (UID: "fbad3465-dc78-4586-a4dc-071709ad50dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:23.536821 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.536798    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-data" pod "fbad3465-dc78-4586-a4dc-071709ad50dd" (UID: "fbad3465-dc78-4586-a4dc-071709ad50dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:23.546824 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.540251    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:23.546824 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.540278    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbad3465-dc78-4586-a4dc-071709ad50dd-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:23.546824 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.540292    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:23.546824 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.540306    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:23.546824 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.540320    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:23.557770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.557412    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-envoy" pod "fbad3465-dc78-4586-a4dc-071709ad50dd" (UID: "fbad3465-dc78-4586-a4dc-071709ad50dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:23.558068 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.558009    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-podinfo" pod "fbad3465-dc78-4586-a4dc-071709ad50dd" (UID: "fbad3465-dc78-4586-a4dc-071709ad50dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:07:23.558142 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.558067    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbad3465-dc78-4586-a4dc-071709ad50dd-kube-api-access-rdtf2" pod "fbad3465-dc78-4586-a4dc-071709ad50dd" (UID: "fbad3465-dc78-4586-a4dc-071709ad50dd"). InnerVolumeSpecName "kube-api-access-rdtf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:23.558470 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.558447    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-token" pod "fbad3465-dc78-4586-a4dc-071709ad50dd" (UID: "fbad3465-dc78-4586-a4dc-071709ad50dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:23.641522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.641492    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:23.641522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.641520    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:23.641759 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.641531    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdtf2\" (UniqueName: \"kubernetes.io/projected/fbad3465-dc78-4586-a4dc-071709ad50dd-kube-api-access-rdtf2\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:23.641759 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.641542    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbad3465-dc78-4586-a4dc-071709ad50dd-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:23.752816 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.752696    6099 patch_prober.go:29] interesting pod/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Oct 30 01:07:23.752816 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:23.752780    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj" podUID="f66406e6-2198-443d-8130-31b590e4f566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Oct 30 01:07:24.302731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:24.302012    6099 patch_prober.go:29] interesting pod/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 30 01:07:24.302731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:24.302070    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8" podUID="7451c167-4bdb-44dd-96ab-ed72782229d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 30 01:07:24.314744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:24.314440    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:24.324742 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:24.323385    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:24.324910 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:24.324747    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:07:24.354273 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:24.354246    6099 patch_prober.go:29] interesting pod/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:07:24.354418 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:24.354291    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t" podUID="fbad3465-dc78-4586-a4dc-071709ad50dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:07:24.423715 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:24.423673    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"
Oct 30 01:07:24.424063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:24.423697    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t" event={"ID":"fbad3465-dc78-4586-a4dc-071709ad50dd","Type":"ContainerDied","Data":"77a4872cb382ccd4e8a37985bdc2018cbb7ee296dc7807f0ba4297b962e15bbe"}
Oct 30 01:07:24.609087 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:24.609005    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"]
Oct 30 01:07:24.613215 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:24.613186    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-43c02d-openshift-default-57c4b85d79-5j69t"]
Oct 30 01:07:24.751303 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:24.751274    6099 patch_prober.go:29] interesting pod/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Oct 30 01:07:24.751481 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:24.751324    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj" podUID="f66406e6-2198-443d-8130-31b590e4f566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Oct 30 01:07:25.305044 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:25.305002    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:25.305498 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:25.305477    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:25.306154 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:25.306133    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:07:25.751255 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:25.751172    6099 patch_prober.go:29] interesting pod/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Oct 30 01:07:25.751255 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:25.751227    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj" podUID="f66406e6-2198-443d-8130-31b590e4f566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Oct 30 01:07:26.264129 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:26.264100    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbad3465-dc78-4586-a4dc-071709ad50dd" path="/var/lib/kubelet/pods/fbad3465-dc78-4586-a4dc-071709ad50dd/volumes"
Oct 30 01:07:26.751190 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:26.751155    6099 patch_prober.go:29] interesting pod/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Oct 30 01:07:26.751382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:26.751208    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj" podUID="f66406e6-2198-443d-8130-31b590e4f566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Oct 30 01:07:27.751038 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:27.751002    6099 patch_prober.go:29] interesting pod/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Oct 30 01:07:27.751233 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:27.751054    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj" podUID="f66406e6-2198-443d-8130-31b590e4f566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Oct 30 01:07:27.871271 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:27.871236    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n"]
Oct 30 01:07:27.904090 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:27.904053    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz"]
Oct 30 01:07:28.755468 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:28.755430    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:28.755976 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:28.755946    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:28.756683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:28.756663    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:07:32.871884 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:32.871829    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n" podUID="84e85af1-92a4-4f4e-8ef3-5cce4c583000" containerName="payload-processing" containerID="cri-o://7d5fa23e632ed13747458c51b661c9cd8cda95e49cc1bed5cae798fa802713de" gracePeriod=25
Oct 30 01:07:32.904355 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:32.904321    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz" podUID="3f9f6975-a3df-4ed1-940a-a801424eec23" containerName="payload-pre-processing" containerID="cri-o://ffcb1f485dacf1b42b6f911f768adb7f4ebb75d544d3a94e4c4bc834f5fefe30" gracePeriod=25
Oct 30 01:07:33.011165 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:33.011124    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc" podUID="ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:07:33.200802 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:33.200770    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n"
Oct 30 01:07:33.238485 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:33.238449    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e85af1-92a4-4f4e-8ef3-5cce4c583000-kube-api-access-d26dc\" (UniqueName: \"kubernetes.io/projected/84e85af1-92a4-4f4e-8ef3-5cce4c583000-kube-api-access-d26dc\") pod \"84e85af1-92a4-4f4e-8ef3-5cce4c583000\" (UID: \"84e85af1-92a4-4f4e-8ef3-5cce4c583000\") "
Oct 30 01:07:33.238675 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:33.238664    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e85af1-92a4-4f4e-8ef3-5cce4c583000-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e85af1-92a4-4f4e-8ef3-5cce4c583000-plugins-config-volume\") pod \"84e85af1-92a4-4f4e-8ef3-5cce4c583000\" (UID: \"84e85af1-92a4-4f4e-8ef3-5cce4c583000\") "
Oct 30 01:07:33.239109 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:33.239080    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e85af1-92a4-4f4e-8ef3-5cce4c583000-plugins-config-volume" pod "84e85af1-92a4-4f4e-8ef3-5cce4c583000" (UID: "84e85af1-92a4-4f4e-8ef3-5cce4c583000"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:33.239631 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:33.239611    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e85af1-92a4-4f4e-8ef3-5cce4c583000-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:33.240651 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:33.240614    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e85af1-92a4-4f4e-8ef3-5cce4c583000-kube-api-access-d26dc" pod "84e85af1-92a4-4f4e-8ef3-5cce4c583000" (UID: "84e85af1-92a4-4f4e-8ef3-5cce4c583000"). InnerVolumeSpecName "kube-api-access-d26dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:33.342110 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:33.342075    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d26dc\" (UniqueName: \"kubernetes.io/projected/84e85af1-92a4-4f4e-8ef3-5cce4c583000-kube-api-access-d26dc\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:33.614416 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:33.614376    6099 generic.go:356] "Generic (PLEG): container finished" podID="84e85af1-92a4-4f4e-8ef3-5cce4c583000" containerID="7d5fa23e632ed13747458c51b661c9cd8cda95e49cc1bed5cae798fa802713de" exitCode=0
Oct 30 01:07:33.614600 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:33.614442    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n"
Oct 30 01:07:33.614600 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:33.614454    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n" event={"ID":"84e85af1-92a4-4f4e-8ef3-5cce4c583000","Type":"ContainerDied","Data":"7d5fa23e632ed13747458c51b661c9cd8cda95e49cc1bed5cae798fa802713de"}
Oct 30 01:07:33.614600 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:33.614531    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n" event={"ID":"84e85af1-92a4-4f4e-8ef3-5cce4c583000","Type":"ContainerDied","Data":"50ba7c465f33b6e76f2be7e6d93798165caa0d461b382e8bd085a2e451d30752"}
Oct 30 01:07:33.614600 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:33.614555    6099 scope.go:122] "RemoveContainer" containerID="7d5fa23e632ed13747458c51b661c9cd8cda95e49cc1bed5cae798fa802713de"
Oct 30 01:07:33.628916 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:33.628897    6099 scope.go:122] "RemoveContainer" containerID="7d5fa23e632ed13747458c51b661c9cd8cda95e49cc1bed5cae798fa802713de"
Oct 30 01:07:33.629220 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:07:33.629192    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d5fa23e632ed13747458c51b661c9cd8cda95e49cc1bed5cae798fa802713de\": container with ID starting with 7d5fa23e632ed13747458c51b661c9cd8cda95e49cc1bed5cae798fa802713de not found: ID does not exist" containerID="7d5fa23e632ed13747458c51b661c9cd8cda95e49cc1bed5cae798fa802713de"
Oct 30 01:07:33.629307 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:33.629232    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d5fa23e632ed13747458c51b661c9cd8cda95e49cc1bed5cae798fa802713de"} err="failed to get container status \"7d5fa23e632ed13747458c51b661c9cd8cda95e49cc1bed5cae798fa802713de\": rpc error: code = NotFound desc = could not find container \"7d5fa23e632ed13747458c51b661c9cd8cda95e49cc1bed5cae798fa802713de\": container with ID starting with 7d5fa23e632ed13747458c51b661c9cd8cda95e49cc1bed5cae798fa802713de not found: ID does not exist"
Oct 30 01:07:33.666495 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:33.666473    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n"]
Oct 30 01:07:33.678199 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:33.678174    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a8b26-5fc6db7f6d-25x6n"]
Oct 30 01:07:34.265631 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:34.265602    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e85af1-92a4-4f4e-8ef3-5cce4c583000" path="/var/lib/kubelet/pods/84e85af1-92a4-4f4e-8ef3-5cce4c583000/volumes"
Oct 30 01:07:34.314908 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:34.314867    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz" podUID="3f9f6975-a3df-4ed1-940a-a801424eec23" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:07:37.834179 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:37.834150    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc"
Oct 30 01:07:37.929847 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:37.929770    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f-plugins-config-volume\") pod \"ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f\" (UID: \"ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f\") "
Oct 30 01:07:37.930006 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:37.929896    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f-kube-api-access-kdjrs\" (UniqueName: \"kubernetes.io/projected/ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f-kube-api-access-kdjrs\") pod \"ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f\" (UID: \"ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f\") "
Oct 30 01:07:37.930239 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:37.930208    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f-plugins-config-volume" pod "ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f" (UID: "ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:37.931051 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:37.931024    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:37.931985 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:37.931945    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f-kube-api-access-kdjrs" pod "ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f" (UID: "ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f"). InnerVolumeSpecName "kube-api-access-kdjrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:38.032423 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:38.032394    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdjrs\" (UniqueName: \"kubernetes.io/projected/ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f-kube-api-access-kdjrs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:38.660339 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:38.660300    6099 generic.go:356] "Generic (PLEG): container finished" podID="ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f" containerID="7984101ec73e2d7e256b925fde5ad2f0dfbae19f94cec37e48e2c445e84c9328" exitCode=0
Oct 30 01:07:38.660517 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:38.660349    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc"
Oct 30 01:07:38.660517 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:38.660384    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc" event={"ID":"ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f","Type":"ContainerDied","Data":"7984101ec73e2d7e256b925fde5ad2f0dfbae19f94cec37e48e2c445e84c9328"}
Oct 30 01:07:38.660517 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:38.660439    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc" event={"ID":"ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f","Type":"ContainerDied","Data":"9503fe54e4b4bb90d95f1e351a50c0e70f0beed950a7140d22aa38578db1f73c"}
Oct 30 01:07:38.660517 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:38.660462    6099 scope.go:122] "RemoveContainer" containerID="7984101ec73e2d7e256b925fde5ad2f0dfbae19f94cec37e48e2c445e84c9328"
Oct 30 01:07:38.671121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:38.670960    6099 scope.go:122] "RemoveContainer" containerID="7984101ec73e2d7e256b925fde5ad2f0dfbae19f94cec37e48e2c445e84c9328"
Oct 30 01:07:38.671237 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:07:38.671215    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7984101ec73e2d7e256b925fde5ad2f0dfbae19f94cec37e48e2c445e84c9328\": container with ID starting with 7984101ec73e2d7e256b925fde5ad2f0dfbae19f94cec37e48e2c445e84c9328 not found: ID does not exist" containerID="7984101ec73e2d7e256b925fde5ad2f0dfbae19f94cec37e48e2c445e84c9328"
Oct 30 01:07:38.671281 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:38.671244    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7984101ec73e2d7e256b925fde5ad2f0dfbae19f94cec37e48e2c445e84c9328"} err="failed to get container status \"7984101ec73e2d7e256b925fde5ad2f0dfbae19f94cec37e48e2c445e84c9328\": rpc error: code = NotFound desc = could not find container \"7984101ec73e2d7e256b925fde5ad2f0dfbae19f94cec37e48e2c445e84c9328\": container with ID starting with 7984101ec73e2d7e256b925fde5ad2f0dfbae19f94cec37e48e2c445e84c9328 not found: ID does not exist"
Oct 30 01:07:38.712441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:38.712418    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc"]
Oct 30 01:07:38.724174 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:38.724152    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a07920-6f78447f99-lzdrc"]
Oct 30 01:07:39.580541 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.580505    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8"]
Oct 30 01:07:39.581404 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.581384    6099 cpu_manager.go:414] "containerMap: removing container" podUID="ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f" containerName="payload-pre-processing"
Oct 30 01:07:39.581456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.581405    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f" containerName="payload-pre-processing"
Oct 30 01:07:39.581456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.581449    6099 cpu_manager.go:414] "containerMap: removing container" podUID="fbad3465-dc78-4586-a4dc-071709ad50dd" containerName="istio-proxy"
Oct 30 01:07:39.581456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.581455    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbad3465-dc78-4586-a4dc-071709ad50dd" containerName="istio-proxy"
Oct 30 01:07:39.581560 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.581474    6099 cpu_manager.go:414] "containerMap: removing container" podUID="84e85af1-92a4-4f4e-8ef3-5cce4c583000" containerName="payload-processing"
Oct 30 01:07:39.581560 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.581479    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e85af1-92a4-4f4e-8ef3-5cce4c583000" containerName="payload-processing"
Oct 30 01:07:39.581620 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.581614    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbad3465-dc78-4586-a4dc-071709ad50dd" containerName="istio-proxy"
Oct 30 01:07:39.581651 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.581627    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e85af1-92a4-4f4e-8ef3-5cce4c583000" containerName="payload-processing"
Oct 30 01:07:39.581651 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.581640    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f" containerName="payload-pre-processing"
Oct 30 01:07:39.589553 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.589527    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8"
Oct 30 01:07:39.607407 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.607344    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8"]
Oct 30 01:07:39.614129 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.614106    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:07:39.614292 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.614236    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:07:39.669107 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.669074    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-820c04-nmml8\" (UID: \"dc93c165-35ad-4bac-9b4a-5c68e4020bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8"
Oct 30 01:07:39.669447 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.669424    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-820c04-nmml8\" (UID: \"dc93c165-35ad-4bac-9b4a-5c68e4020bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8"
Oct 30 01:07:39.669887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.669863    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q27cn\" (UniqueName: \"kubernetes.io/projected/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-kube-api-access-q27cn\") pod \"maas-api-revoke-keys-e2e-inf-a-820c04-nmml8\" (UID: \"dc93c165-35ad-4bac-9b4a-5c68e4020bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8"
Oct 30 01:07:39.771471 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.771432    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q27cn\" (UniqueName: \"kubernetes.io/projected/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-kube-api-access-q27cn\") pod \"maas-api-revoke-keys-e2e-inf-a-820c04-nmml8\" (UID: \"dc93c165-35ad-4bac-9b4a-5c68e4020bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8"
Oct 30 01:07:39.771644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.771582    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-820c04-nmml8\" (UID: \"dc93c165-35ad-4bac-9b4a-5c68e4020bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8"
Oct 30 01:07:39.771644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.771611    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-820c04-nmml8\" (UID: \"dc93c165-35ad-4bac-9b4a-5c68e4020bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8"
Oct 30 01:07:39.772327 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.772301    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-820c04-nmml8\" (UID: \"dc93c165-35ad-4bac-9b4a-5c68e4020bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8"
Oct 30 01:07:39.773929 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.773904    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-820c04-nmml8\" (UID: \"dc93c165-35ad-4bac-9b4a-5c68e4020bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8"
Oct 30 01:07:39.793979 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.793927    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q27cn\" (UniqueName: \"kubernetes.io/projected/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-kube-api-access-q27cn\") pod \"maas-api-revoke-keys-e2e-inf-a-820c04-nmml8\" (UID: \"dc93c165-35ad-4bac-9b4a-5c68e4020bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8"
Oct 30 01:07:39.912193 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:39.912115    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8"
Oct 30 01:07:40.274090 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:07:40.274037    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc93c165_35ad_4bac_9b4a_5c68e4020bfa.slice/crio-cf266c1e301ba400ac3e02dd4aa1da67f793c925dcfa97f41fc8c12c8def559e WatchSource:0}: Error finding container cf266c1e301ba400ac3e02dd4aa1da67f793c925dcfa97f41fc8c12c8def559e: Status 404 returned error can't find the container with id cf266c1e301ba400ac3e02dd4aa1da67f793c925dcfa97f41fc8c12c8def559e
Oct 30 01:07:40.299502 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:40.299468    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f" path="/var/lib/kubelet/pods/ab72b6bc-2c27-4954-b9c3-b26b7fc1f43f/volumes"
Oct 30 01:07:40.300225 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:40.300203    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8"]
Oct 30 01:07:40.684548 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:40.684515    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8" event={"ID":"dc93c165-35ad-4bac-9b4a-5c68e4020bfa","Type":"ContainerStarted","Data":"cf266c1e301ba400ac3e02dd4aa1da67f793c925dcfa97f41fc8c12c8def559e"}
Oct 30 01:07:41.694231 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:41.694144    6099 generic.go:356] "Generic (PLEG): container finished" podID="dc93c165-35ad-4bac-9b4a-5c68e4020bfa" containerID="dfc021b2f376d2310a547237a431723cfd022efd87812c725fe9b476e1b149aa" exitCode=0
Oct 30 01:07:41.694231 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:41.694207    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8" event={"ID":"dc93c165-35ad-4bac-9b4a-5c68e4020bfa","Type":"ContainerDied","Data":"dfc021b2f376d2310a547237a431723cfd022efd87812c725fe9b476e1b149aa"}
Oct 30 01:07:42.872906 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:42.872874    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8"
Oct 30 01:07:43.026414 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:43.026340    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-maas-api-xks-ca\") pod \"dc93c165-35ad-4bac-9b4a-5c68e4020bfa\" (UID: \"dc93c165-35ad-4bac-9b4a-5c68e4020bfa\") "
Oct 30 01:07:43.026570 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:43.026447    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-kube-api-access-q27cn\" (UniqueName: \"kubernetes.io/projected/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-kube-api-access-q27cn\") pod \"dc93c165-35ad-4bac-9b4a-5c68e4020bfa\" (UID: \"dc93c165-35ad-4bac-9b4a-5c68e4020bfa\") "
Oct 30 01:07:43.026570 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:43.026490    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-maas-api-openshift-ca\") pod \"dc93c165-35ad-4bac-9b4a-5c68e4020bfa\" (UID: \"dc93c165-35ad-4bac-9b4a-5c68e4020bfa\") "
Oct 30 01:07:43.027000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:43.026969    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-maas-api-openshift-ca" pod "dc93c165-35ad-4bac-9b4a-5c68e4020bfa" (UID: "dc93c165-35ad-4bac-9b4a-5c68e4020bfa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:43.027504 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:43.027481    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.028674 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:43.028641    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-kube-api-access-q27cn" pod "dc93c165-35ad-4bac-9b4a-5c68e4020bfa" (UID: "dc93c165-35ad-4bac-9b4a-5c68e4020bfa"). InnerVolumeSpecName "kube-api-access-q27cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:43.028674 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:43.028642    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-maas-api-xks-ca" pod "dc93c165-35ad-4bac-9b4a-5c68e4020bfa" (UID: "dc93c165-35ad-4bac-9b4a-5c68e4020bfa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:07:43.129453 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:43.129421    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.129453 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:43.129449    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q27cn\" (UniqueName: \"kubernetes.io/projected/dc93c165-35ad-4bac-9b4a-5c68e4020bfa-kube-api-access-q27cn\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:43.712570 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:43.712539    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8"
Oct 30 01:07:43.712570 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:43.712556    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8" event={"ID":"dc93c165-35ad-4bac-9b4a-5c68e4020bfa","Type":"ContainerDied","Data":"cf266c1e301ba400ac3e02dd4aa1da67f793c925dcfa97f41fc8c12c8def559e"}
Oct 30 01:07:43.712827 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:43.712608    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf266c1e301ba400ac3e02dd4aa1da67f793c925dcfa97f41fc8c12c8def559e"
Oct 30 01:07:44.316399 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:44.316366    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz" podUID="3f9f6975-a3df-4ed1-940a-a801424eec23" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:07:46.630970 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:07:46.630932    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:07:50.026817 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:50.026781    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd"]
Oct 30 01:07:50.066832 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:50.066800    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd"]
Oct 30 01:07:52.179698 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:52.179662    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"]
Oct 30 01:07:52.180188 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:52.179912    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh" podUID="c789663e-00c2-4c17-9b12-784459a7a9e4" containerName="maas-api" containerID="cri-o://6f35717d8a81bacb31738c795694a9b52915457328b1ab3248ef4ec8cdd4b343" gracePeriod=30
Oct 30 01:07:52.872242 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:52.872086    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"
Oct 30 01:07:52.886039 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:52.885999    6099 generic.go:356] "Generic (PLEG): container finished" podID="c789663e-00c2-4c17-9b12-784459a7a9e4" containerID="6f35717d8a81bacb31738c795694a9b52915457328b1ab3248ef4ec8cdd4b343" exitCode=0
Oct 30 01:07:52.886226 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:52.886060    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"
Oct 30 01:07:52.886226 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:52.886152    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh" event={"ID":"c789663e-00c2-4c17-9b12-784459a7a9e4","Type":"ContainerDied","Data":"6f35717d8a81bacb31738c795694a9b52915457328b1ab3248ef4ec8cdd4b343"}
Oct 30 01:07:52.886553 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:52.886244    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh" event={"ID":"c789663e-00c2-4c17-9b12-784459a7a9e4","Type":"ContainerDied","Data":"b9961b2ff1939bfb2775bcd68987e24364ec0c0ce4f54f8cb9f32c04a891d984"}
Oct 30 01:07:52.886553 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:52.886275    6099 scope.go:122] "RemoveContainer" containerID="6f35717d8a81bacb31738c795694a9b52915457328b1ab3248ef4ec8cdd4b343"
Oct 30 01:07:52.899067 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:52.899040    6099 scope.go:122] "RemoveContainer" containerID="6f35717d8a81bacb31738c795694a9b52915457328b1ab3248ef4ec8cdd4b343"
Oct 30 01:07:52.899394 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:07:52.899370    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f35717d8a81bacb31738c795694a9b52915457328b1ab3248ef4ec8cdd4b343\": container with ID starting with 6f35717d8a81bacb31738c795694a9b52915457328b1ab3248ef4ec8cdd4b343 not found: ID does not exist" containerID="6f35717d8a81bacb31738c795694a9b52915457328b1ab3248ef4ec8cdd4b343"
Oct 30 01:07:52.899489 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:52.899403    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f35717d8a81bacb31738c795694a9b52915457328b1ab3248ef4ec8cdd4b343"} err="failed to get container status \"6f35717d8a81bacb31738c795694a9b52915457328b1ab3248ef4ec8cdd4b343\": rpc error: code = NotFound desc = could not find container \"6f35717d8a81bacb31738c795694a9b52915457328b1ab3248ef4ec8cdd4b343\": container with ID starting with 6f35717d8a81bacb31738c795694a9b52915457328b1ab3248ef4ec8cdd4b343 not found: ID does not exist"
Oct 30 01:07:53.006115 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.006082    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c789663e-00c2-4c17-9b12-784459a7a9e4-kube-api-access-c5xg8\" (UniqueName: \"kubernetes.io/projected/c789663e-00c2-4c17-9b12-784459a7a9e4-kube-api-access-c5xg8\") pod \"c789663e-00c2-4c17-9b12-784459a7a9e4\" (UID: \"c789663e-00c2-4c17-9b12-784459a7a9e4\") "
Oct 30 01:07:53.006278 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.006168    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c789663e-00c2-4c17-9b12-784459a7a9e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c789663e-00c2-4c17-9b12-784459a7a9e4-maas-api-tls\") pod \"c789663e-00c2-4c17-9b12-784459a7a9e4\" (UID: \"c789663e-00c2-4c17-9b12-784459a7a9e4\") "
Oct 30 01:07:53.006278 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.006253    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c789663e-00c2-4c17-9b12-784459a7a9e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c789663e-00c2-4c17-9b12-784459a7a9e4-openshift-service-ca\") pod \"c789663e-00c2-4c17-9b12-784459a7a9e4\" (UID: \"c789663e-00c2-4c17-9b12-784459a7a9e4\") "
Oct 30 01:07:53.006414 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.006402    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c789663e-00c2-4c17-9b12-784459a7a9e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c789663e-00c2-4c17-9b12-784459a7a9e4-metrics-certs\") pod \"c789663e-00c2-4c17-9b12-784459a7a9e4\" (UID: \"c789663e-00c2-4c17-9b12-784459a7a9e4\") "
Oct 30 01:07:53.006621 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.006597    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c789663e-00c2-4c17-9b12-784459a7a9e4-openshift-service-ca" pod "c789663e-00c2-4c17-9b12-784459a7a9e4" (UID: "c789663e-00c2-4c17-9b12-784459a7a9e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:53.007169 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.007146    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c789663e-00c2-4c17-9b12-784459a7a9e4-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:53.008518 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.008474    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c789663e-00c2-4c17-9b12-784459a7a9e4-metrics-certs" pod "c789663e-00c2-4c17-9b12-784459a7a9e4" (UID: "c789663e-00c2-4c17-9b12-784459a7a9e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:07:53.008603 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.008576    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c789663e-00c2-4c17-9b12-784459a7a9e4-maas-api-tls" pod "c789663e-00c2-4c17-9b12-784459a7a9e4" (UID: "c789663e-00c2-4c17-9b12-784459a7a9e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:07:53.008735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.008682    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c789663e-00c2-4c17-9b12-784459a7a9e4-kube-api-access-c5xg8" pod "c789663e-00c2-4c17-9b12-784459a7a9e4" (UID: "c789663e-00c2-4c17-9b12-784459a7a9e4"). InnerVolumeSpecName "kube-api-access-c5xg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:53.112340 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.111769    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c789663e-00c2-4c17-9b12-784459a7a9e4-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:53.112340 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.111803    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5xg8\" (UniqueName: \"kubernetes.io/projected/c789663e-00c2-4c17-9b12-784459a7a9e4-kube-api-access-c5xg8\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:53.112340 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.111818    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c789663e-00c2-4c17-9b12-784459a7a9e4-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:53.256820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.256797    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz"
Oct 30 01:07:53.290400 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.290360    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"]
Oct 30 01:07:53.301946 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.301921    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a07920-5b7bb949d5-zwhfh"]
Oct 30 01:07:53.314952 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.314924    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9f6975-a3df-4ed1-940a-a801424eec23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f9f6975-a3df-4ed1-940a-a801424eec23-plugins-config-volume\") pod \"3f9f6975-a3df-4ed1-940a-a801424eec23\" (UID: \"3f9f6975-a3df-4ed1-940a-a801424eec23\") "
Oct 30 01:07:53.315084 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.314970    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9f6975-a3df-4ed1-940a-a801424eec23-kube-api-access-zwf2b\" (UniqueName: \"kubernetes.io/projected/3f9f6975-a3df-4ed1-940a-a801424eec23-kube-api-access-zwf2b\") pod \"3f9f6975-a3df-4ed1-940a-a801424eec23\" (UID: \"3f9f6975-a3df-4ed1-940a-a801424eec23\") "
Oct 30 01:07:53.315265 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.315245    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9f6975-a3df-4ed1-940a-a801424eec23-plugins-config-volume" pod "3f9f6975-a3df-4ed1-940a-a801424eec23" (UID: "3f9f6975-a3df-4ed1-940a-a801424eec23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:53.316156 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.316132    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f9f6975-a3df-4ed1-940a-a801424eec23-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:53.317289 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.317264    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9f6975-a3df-4ed1-940a-a801424eec23-kube-api-access-zwf2b" pod "3f9f6975-a3df-4ed1-940a-a801424eec23" (UID: "3f9f6975-a3df-4ed1-940a-a801424eec23"). InnerVolumeSpecName "kube-api-access-zwf2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:53.418358 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.418319    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwf2b\" (UniqueName: \"kubernetes.io/projected/3f9f6975-a3df-4ed1-940a-a801424eec23-kube-api-access-zwf2b\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:53.897343 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.897309    6099 generic.go:356] "Generic (PLEG): container finished" podID="3f9f6975-a3df-4ed1-940a-a801424eec23" containerID="ffcb1f485dacf1b42b6f911f768adb7f4ebb75d544d3a94e4c4bc834f5fefe30" exitCode=0
Oct 30 01:07:53.897547 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.897346    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz"
Oct 30 01:07:53.897547 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.897389    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz" event={"ID":"3f9f6975-a3df-4ed1-940a-a801424eec23","Type":"ContainerDied","Data":"ffcb1f485dacf1b42b6f911f768adb7f4ebb75d544d3a94e4c4bc834f5fefe30"}
Oct 30 01:07:53.897547 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.897451    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz" event={"ID":"3f9f6975-a3df-4ed1-940a-a801424eec23","Type":"ContainerDied","Data":"eb94b09e9afea31cd1438b644f6c0b37490270d20e230ed170ce52d5ac859e41"}
Oct 30 01:07:53.897547 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.897480    6099 scope.go:122] "RemoveContainer" containerID="ffcb1f485dacf1b42b6f911f768adb7f4ebb75d544d3a94e4c4bc834f5fefe30"
Oct 30 01:07:53.909643 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.909618    6099 scope.go:122] "RemoveContainer" containerID="ffcb1f485dacf1b42b6f911f768adb7f4ebb75d544d3a94e4c4bc834f5fefe30"
Oct 30 01:07:53.909908 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:07:53.909891    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffcb1f485dacf1b42b6f911f768adb7f4ebb75d544d3a94e4c4bc834f5fefe30\": container with ID starting with ffcb1f485dacf1b42b6f911f768adb7f4ebb75d544d3a94e4c4bc834f5fefe30 not found: ID does not exist" containerID="ffcb1f485dacf1b42b6f911f768adb7f4ebb75d544d3a94e4c4bc834f5fefe30"
Oct 30 01:07:53.909958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.909919    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffcb1f485dacf1b42b6f911f768adb7f4ebb75d544d3a94e4c4bc834f5fefe30"} err="failed to get container status \"ffcb1f485dacf1b42b6f911f768adb7f4ebb75d544d3a94e4c4bc834f5fefe30\": rpc error: code = NotFound desc = could not find container \"ffcb1f485dacf1b42b6f911f768adb7f4ebb75d544d3a94e4c4bc834f5fefe30\": container with ID starting with ffcb1f485dacf1b42b6f911f768adb7f4ebb75d544d3a94e4c4bc834f5fefe30 not found: ID does not exist"
Oct 30 01:07:53.948632 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.948596    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz"]
Oct 30 01:07:53.960424 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:53.960403    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a8b26-857bc754ff-kvmnz"]
Oct 30 01:07:54.264286 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:54.264214    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f9f6975-a3df-4ed1-940a-a801424eec23" path="/var/lib/kubelet/pods/3f9f6975-a3df-4ed1-940a-a801424eec23/volumes"
Oct 30 01:07:54.264854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:54.264835    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c789663e-00c2-4c17-9b12-784459a7a9e4" path="/var/lib/kubelet/pods/c789663e-00c2-4c17-9b12-784459a7a9e4/volumes"
Oct 30 01:07:55.027581 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:55.027538    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd" podUID="15ed7bd6-34b8-49f2-af6b-453b0c7f921a" containerName="payload-processing" containerID="cri-o://f3cc288764ab2d5194bcba6407fd595cd0826cf2a8fca50a4129504644f9ac26" gracePeriod=25
Oct 30 01:07:55.068053 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:55.068020    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd" podUID="66824435-d248-47f0-ab98-f8f2c53e25b1" containerName="payload-pre-processing" containerID="cri-o://c1198035f1cbb5fa1a4e781a9437167344d573b84a79bc5913e84c011c21e342" gracePeriod=25
Oct 30 01:07:55.359592 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:55.359566    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd"
Oct 30 01:07:55.452550 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:55.452149    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15ed7bd6-34b8-49f2-af6b-453b0c7f921a-kube-api-access-4xx8h\" (UniqueName: \"kubernetes.io/projected/15ed7bd6-34b8-49f2-af6b-453b0c7f921a-kube-api-access-4xx8h\") pod \"15ed7bd6-34b8-49f2-af6b-453b0c7f921a\" (UID: \"15ed7bd6-34b8-49f2-af6b-453b0c7f921a\") "
Oct 30 01:07:55.452550 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:55.452394    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15ed7bd6-34b8-49f2-af6b-453b0c7f921a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15ed7bd6-34b8-49f2-af6b-453b0c7f921a-plugins-config-volume\") pod \"15ed7bd6-34b8-49f2-af6b-453b0c7f921a\" (UID: \"15ed7bd6-34b8-49f2-af6b-453b0c7f921a\") "
Oct 30 01:07:55.452904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:55.452875    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ed7bd6-34b8-49f2-af6b-453b0c7f921a-plugins-config-volume" pod "15ed7bd6-34b8-49f2-af6b-453b0c7f921a" (UID: "15ed7bd6-34b8-49f2-af6b-453b0c7f921a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:55.453353 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:55.453328    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15ed7bd6-34b8-49f2-af6b-453b0c7f921a-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:55.454960 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:55.454926    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ed7bd6-34b8-49f2-af6b-453b0c7f921a-kube-api-access-4xx8h" pod "15ed7bd6-34b8-49f2-af6b-453b0c7f921a" (UID: "15ed7bd6-34b8-49f2-af6b-453b0c7f921a"). InnerVolumeSpecName "kube-api-access-4xx8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:55.554869 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:55.554837    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xx8h\" (UniqueName: \"kubernetes.io/projected/15ed7bd6-34b8-49f2-af6b-453b0c7f921a-kube-api-access-4xx8h\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:07:55.915700 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:55.915666    6099 generic.go:356] "Generic (PLEG): container finished" podID="15ed7bd6-34b8-49f2-af6b-453b0c7f921a" containerID="f3cc288764ab2d5194bcba6407fd595cd0826cf2a8fca50a4129504644f9ac26" exitCode=0
Oct 30 01:07:55.915907 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:55.915739    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd"
Oct 30 01:07:55.915907 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:55.915741    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd" event={"ID":"15ed7bd6-34b8-49f2-af6b-453b0c7f921a","Type":"ContainerDied","Data":"f3cc288764ab2d5194bcba6407fd595cd0826cf2a8fca50a4129504644f9ac26"}
Oct 30 01:07:55.915907 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:55.915854    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd" event={"ID":"15ed7bd6-34b8-49f2-af6b-453b0c7f921a","Type":"ContainerDied","Data":"db9adb8f240ca597d2a6e65f9a1a2bf7cfe034da4a94749f496977cfd4da522b"}
Oct 30 01:07:55.915907 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:55.915876    6099 scope.go:122] "RemoveContainer" containerID="f3cc288764ab2d5194bcba6407fd595cd0826cf2a8fca50a4129504644f9ac26"
Oct 30 01:07:55.927266 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:55.927243    6099 scope.go:122] "RemoveContainer" containerID="f3cc288764ab2d5194bcba6407fd595cd0826cf2a8fca50a4129504644f9ac26"
Oct 30 01:07:55.927511 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:07:55.927491    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3cc288764ab2d5194bcba6407fd595cd0826cf2a8fca50a4129504644f9ac26\": container with ID starting with f3cc288764ab2d5194bcba6407fd595cd0826cf2a8fca50a4129504644f9ac26 not found: ID does not exist" containerID="f3cc288764ab2d5194bcba6407fd595cd0826cf2a8fca50a4129504644f9ac26"
Oct 30 01:07:55.927566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:55.927517    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3cc288764ab2d5194bcba6407fd595cd0826cf2a8fca50a4129504644f9ac26"} err="failed to get container status \"f3cc288764ab2d5194bcba6407fd595cd0826cf2a8fca50a4129504644f9ac26\": rpc error: code = NotFound desc = could not find container \"f3cc288764ab2d5194bcba6407fd595cd0826cf2a8fca50a4129504644f9ac26\": container with ID starting with f3cc288764ab2d5194bcba6407fd595cd0826cf2a8fca50a4129504644f9ac26 not found: ID does not exist"
Oct 30 01:07:55.967208 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:55.967179    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd"]
Oct 30 01:07:55.978814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:55.978783    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-820c04-7b94544c4b-5sgbd"]
Oct 30 01:07:56.263164 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:56.263089    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15ed7bd6-34b8-49f2-af6b-453b0c7f921a" path="/var/lib/kubelet/pods/15ed7bd6-34b8-49f2-af6b-453b0c7f921a/volumes"
Oct 30 01:07:59.921363 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:07:59.921326    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd" podUID="66824435-d248-47f0-ab98-f8f2c53e25b1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:08:00.160508 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:00.160471    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"]
Oct 30 01:08:00.161077 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:00.161046    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48" podUID="f85bad1f-83ed-42b6-b88a-c1388b5480b2" containerName="istio-proxy" containerID="cri-o://16c29f97a01515549fb10b641f313d731bf64b45a0ad1f8b0a3cd67002e2de39" gracePeriod=30
Oct 30 01:08:04.807655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:04.807620    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"]
Oct 30 01:08:04.808099 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:04.807821    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46" podUID="261c5da2-a187-4ad5-876f-35db0c631a06" containerName="maas-api" containerID="cri-o://8266ea777fc0426ee26eada0ed52cf4bb6ca09c71d990ba92b0e7986a671723f" gracePeriod=30
Oct 30 01:08:05.011512 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.011472    6099 generic.go:356] "Generic (PLEG): container finished" podID="261c5da2-a187-4ad5-876f-35db0c631a06" containerID="8266ea777fc0426ee26eada0ed52cf4bb6ca09c71d990ba92b0e7986a671723f" exitCode=0
Oct 30 01:08:05.011762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.011694    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46" event={"ID":"261c5da2-a187-4ad5-876f-35db0c631a06","Type":"ContainerDied","Data":"8266ea777fc0426ee26eada0ed52cf4bb6ca09c71d990ba92b0e7986a671723f"}
Oct 30 01:08:05.235295 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.234905    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"
Oct 30 01:08:05.337727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.327002    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/261c5da2-a187-4ad5-876f-35db0c631a06-metrics-certs\" (UniqueName: \"kubernetes.io/secret/261c5da2-a187-4ad5-876f-35db0c631a06-metrics-certs\") pod \"261c5da2-a187-4ad5-876f-35db0c631a06\" (UID: \"261c5da2-a187-4ad5-876f-35db0c631a06\") "
Oct 30 01:08:05.337727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.327627    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/261c5da2-a187-4ad5-876f-35db0c631a06-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/261c5da2-a187-4ad5-876f-35db0c631a06-maas-api-tls\") pod \"261c5da2-a187-4ad5-876f-35db0c631a06\" (UID: \"261c5da2-a187-4ad5-876f-35db0c631a06\") "
Oct 30 01:08:05.337727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.328464    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/261c5da2-a187-4ad5-876f-35db0c631a06-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/261c5da2-a187-4ad5-876f-35db0c631a06-openshift-service-ca\") pod \"261c5da2-a187-4ad5-876f-35db0c631a06\" (UID: \"261c5da2-a187-4ad5-876f-35db0c631a06\") "
Oct 30 01:08:05.337727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.328892    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261c5da2-a187-4ad5-876f-35db0c631a06-kube-api-access-t4z2f\" (UniqueName: \"kubernetes.io/projected/261c5da2-a187-4ad5-876f-35db0c631a06-kube-api-access-t4z2f\") pod \"261c5da2-a187-4ad5-876f-35db0c631a06\" (UID: \"261c5da2-a187-4ad5-876f-35db0c631a06\") "
Oct 30 01:08:05.342725 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.339127    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261c5da2-a187-4ad5-876f-35db0c631a06-openshift-service-ca" pod "261c5da2-a187-4ad5-876f-35db0c631a06" (UID: "261c5da2-a187-4ad5-876f-35db0c631a06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:05.402803 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.385559    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/261c5da2-a187-4ad5-876f-35db0c631a06-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:05.452818 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.449950    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261c5da2-a187-4ad5-876f-35db0c631a06-metrics-certs" pod "261c5da2-a187-4ad5-876f-35db0c631a06" (UID: "261c5da2-a187-4ad5-876f-35db0c631a06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:05.519485 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.519451    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/261c5da2-a187-4ad5-876f-35db0c631a06-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:05.544732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.523787    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261c5da2-a187-4ad5-876f-35db0c631a06-kube-api-access-t4z2f" pod "261c5da2-a187-4ad5-876f-35db0c631a06" (UID: "261c5da2-a187-4ad5-876f-35db0c631a06"). InnerVolumeSpecName "kube-api-access-t4z2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:05.544732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.531954    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261c5da2-a187-4ad5-876f-35db0c631a06-maas-api-tls" pod "261c5da2-a187-4ad5-876f-35db0c631a06" (UID: "261c5da2-a187-4ad5-876f-35db0c631a06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:05.621899 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.621871    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4z2f\" (UniqueName: \"kubernetes.io/projected/261c5da2-a187-4ad5-876f-35db0c631a06-kube-api-access-t4z2f\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:05.622033 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.622021    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/261c5da2-a187-4ad5-876f-35db0c631a06-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:05.740792 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.740754    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:08:05.836603 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.836558    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-podinfo\") pod \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") "
Oct 30 01:08:05.837123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.836777    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-credential-socket\") pod \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") "
Oct 30 01:08:05.837123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.836851    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-workload-certs\") pod \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") "
Oct 30 01:08:05.837123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.836896    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istiod-ca-cert\") pod \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") "
Oct 30 01:08:05.837123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.837050    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-workload-socket\") pod \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") "
Oct 30 01:08:05.837123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.837097    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-credential-socket" pod "f85bad1f-83ed-42b6-b88a-c1388b5480b2" (UID: "f85bad1f-83ed-42b6-b88a-c1388b5480b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:05.837653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.837177    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-envoy\") pod \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") "
Oct 30 01:08:05.837653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.837276    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-data\") pod \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") "
Oct 30 01:08:05.837653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.837325    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-workload-certs" pod "f85bad1f-83ed-42b6-b88a-c1388b5480b2" (UID: "f85bad1f-83ed-42b6-b88a-c1388b5480b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:05.837653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.837569    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85bad1f-83ed-42b6-b88a-c1388b5480b2-kube-api-access-hlrtb\" (UniqueName: \"kubernetes.io/projected/f85bad1f-83ed-42b6-b88a-c1388b5480b2-kube-api-access-hlrtb\") pod \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") "
Oct 30 01:08:05.837653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.837572    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-workload-socket" pod "f85bad1f-83ed-42b6-b88a-c1388b5480b2" (UID: "f85bad1f-83ed-42b6-b88a-c1388b5480b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:05.838107 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.837952    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istiod-ca-cert" pod "f85bad1f-83ed-42b6-b88a-c1388b5480b2" (UID: "f85bad1f-83ed-42b6-b88a-c1388b5480b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:05.838107 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.837996    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-token\" (UniqueName: \"kubernetes.io/projected/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-token\") pod \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\" (UID: \"f85bad1f-83ed-42b6-b88a-c1388b5480b2\") "
Oct 30 01:08:05.839099 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.838961    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:05.839099 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.838991    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:05.839099 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.839007    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:05.839099 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.839019    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:05.844493 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.840372    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-envoy" pod "f85bad1f-83ed-42b6-b88a-c1388b5480b2" (UID: "f85bad1f-83ed-42b6-b88a-c1388b5480b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:05.844493 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.840812    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-data" pod "f85bad1f-83ed-42b6-b88a-c1388b5480b2" (UID: "f85bad1f-83ed-42b6-b88a-c1388b5480b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:05.844493 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.842004    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-podinfo" pod "f85bad1f-83ed-42b6-b88a-c1388b5480b2" (UID: "f85bad1f-83ed-42b6-b88a-c1388b5480b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:08:05.847448 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.847418    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-token" pod "f85bad1f-83ed-42b6-b88a-c1388b5480b2" (UID: "f85bad1f-83ed-42b6-b88a-c1388b5480b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:05.849874 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.849849    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85bad1f-83ed-42b6-b88a-c1388b5480b2-kube-api-access-hlrtb" pod "f85bad1f-83ed-42b6-b88a-c1388b5480b2" (UID: "f85bad1f-83ed-42b6-b88a-c1388b5480b2"). InnerVolumeSpecName "kube-api-access-hlrtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:05.940412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.940381    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:05.940412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.940408    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:05.940412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.940417    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlrtb\" (UniqueName: \"kubernetes.io/projected/f85bad1f-83ed-42b6-b88a-c1388b5480b2-kube-api-access-hlrtb\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:05.940643 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.940428    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:05.940643 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:05.940437    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85bad1f-83ed-42b6-b88a-c1388b5480b2-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:06.027460 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.027377    6099 generic.go:356] "Generic (PLEG): container finished" podID="f85bad1f-83ed-42b6-b88a-c1388b5480b2" containerID="16c29f97a01515549fb10b641f313d731bf64b45a0ad1f8b0a3cd67002e2de39" exitCode=0
Oct 30 01:08:06.027660 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.027639    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48" event={"ID":"f85bad1f-83ed-42b6-b88a-c1388b5480b2","Type":"ContainerDied","Data":"16c29f97a01515549fb10b641f313d731bf64b45a0ad1f8b0a3cd67002e2de39"}
Oct 30 01:08:06.027781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.027698    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48" event={"ID":"f85bad1f-83ed-42b6-b88a-c1388b5480b2","Type":"ContainerDied","Data":"9bd0b63a06da77093a97d940611d6b571ccf8028bc2ce2061136121d235d0047"}
Oct 30 01:08:06.027781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.027748    6099 scope.go:122] "RemoveContainer" containerID="16c29f97a01515549fb10b641f313d731bf64b45a0ad1f8b0a3cd67002e2de39"
Oct 30 01:08:06.027922 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.027907    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"
Oct 30 01:08:06.034666 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.034597    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46" event={"ID":"261c5da2-a187-4ad5-876f-35db0c631a06","Type":"ContainerDied","Data":"838d80f565575b2dcadb5645b02e21fad0118659f8baed647af1614b87875a1a"}
Oct 30 01:08:06.034666 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.034610    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"
Oct 30 01:08:06.047137 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.047118    6099 scope.go:122] "RemoveContainer" containerID="a1f83ceeed3b9e9fcedc6c53c3b8dea1ca41a624a2b71466bf8ce695c80c77fa"
Oct 30 01:08:06.057575 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.057556    6099 scope.go:122] "RemoveContainer" containerID="16c29f97a01515549fb10b641f313d731bf64b45a0ad1f8b0a3cd67002e2de39"
Oct 30 01:08:06.057864 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:08:06.057842    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16c29f97a01515549fb10b641f313d731bf64b45a0ad1f8b0a3cd67002e2de39\": container with ID starting with 16c29f97a01515549fb10b641f313d731bf64b45a0ad1f8b0a3cd67002e2de39 not found: ID does not exist" containerID="16c29f97a01515549fb10b641f313d731bf64b45a0ad1f8b0a3cd67002e2de39"
Oct 30 01:08:06.057948 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.057872    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16c29f97a01515549fb10b641f313d731bf64b45a0ad1f8b0a3cd67002e2de39"} err="failed to get container status \"16c29f97a01515549fb10b641f313d731bf64b45a0ad1f8b0a3cd67002e2de39\": rpc error: code = NotFound desc = could not find container \"16c29f97a01515549fb10b641f313d731bf64b45a0ad1f8b0a3cd67002e2de39\": container with ID starting with 16c29f97a01515549fb10b641f313d731bf64b45a0ad1f8b0a3cd67002e2de39 not found: ID does not exist"
Oct 30 01:08:06.057948 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.057890    6099 scope.go:122] "RemoveContainer" containerID="a1f83ceeed3b9e9fcedc6c53c3b8dea1ca41a624a2b71466bf8ce695c80c77fa"
Oct 30 01:08:06.058154 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:08:06.058135    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f83ceeed3b9e9fcedc6c53c3b8dea1ca41a624a2b71466bf8ce695c80c77fa\": container with ID starting with a1f83ceeed3b9e9fcedc6c53c3b8dea1ca41a624a2b71466bf8ce695c80c77fa not found: ID does not exist" containerID="a1f83ceeed3b9e9fcedc6c53c3b8dea1ca41a624a2b71466bf8ce695c80c77fa"
Oct 30 01:08:06.058231 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.058159    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f83ceeed3b9e9fcedc6c53c3b8dea1ca41a624a2b71466bf8ce695c80c77fa"} err="failed to get container status \"a1f83ceeed3b9e9fcedc6c53c3b8dea1ca41a624a2b71466bf8ce695c80c77fa\": rpc error: code = NotFound desc = could not find container \"a1f83ceeed3b9e9fcedc6c53c3b8dea1ca41a624a2b71466bf8ce695c80c77fa\": container with ID starting with a1f83ceeed3b9e9fcedc6c53c3b8dea1ca41a624a2b71466bf8ce695c80c77fa not found: ID does not exist"
Oct 30 01:08:06.058231 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.058172    6099 scope.go:122] "RemoveContainer" containerID="8266ea777fc0426ee26eada0ed52cf4bb6ca09c71d990ba92b0e7986a671723f"
Oct 30 01:08:06.136378 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.136339    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"]
Oct 30 01:08:06.146042 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.146008    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a07920-openshift-default-5f4bd748c-pcb48"]
Oct 30 01:08:06.166225 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.166190    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"]
Oct 30 01:08:06.172236 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.172208    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a8b26-86df449c85-4mz46"]
Oct 30 01:08:06.264952 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.264923    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261c5da2-a187-4ad5-876f-35db0c631a06" path="/var/lib/kubelet/pods/261c5da2-a187-4ad5-876f-35db0c631a06/volumes"
Oct 30 01:08:06.265427 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:06.265408    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f85bad1f-83ed-42b6-b88a-c1388b5480b2" path="/var/lib/kubelet/pods/f85bad1f-83ed-42b6-b88a-c1388b5480b2/volumes"
Oct 30 01:08:07.210812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.210780    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h"]
Oct 30 01:08:07.211541 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211524    6099 cpu_manager.go:414] "containerMap: removing container" podUID="f85bad1f-83ed-42b6-b88a-c1388b5480b2" containerName="istio-proxy"
Oct 30 01:08:07.211541 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211540    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85bad1f-83ed-42b6-b88a-c1388b5480b2" containerName="istio-proxy"
Oct 30 01:08:07.211695 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211552    6099 cpu_manager.go:414] "containerMap: removing container" podUID="dc93c165-35ad-4bac-9b4a-5c68e4020bfa" containerName="revoke-keys"
Oct 30 01:08:07.211695 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211558    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc93c165-35ad-4bac-9b4a-5c68e4020bfa" containerName="revoke-keys"
Oct 30 01:08:07.211695 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211570    6099 cpu_manager.go:414] "containerMap: removing container" podUID="3f9f6975-a3df-4ed1-940a-a801424eec23" containerName="payload-pre-processing"
Oct 30 01:08:07.211695 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211577    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9f6975-a3df-4ed1-940a-a801424eec23" containerName="payload-pre-processing"
Oct 30 01:08:07.211695 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211602    6099 cpu_manager.go:414] "containerMap: removing container" podUID="261c5da2-a187-4ad5-876f-35db0c631a06" containerName="maas-api"
Oct 30 01:08:07.211695 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211608    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261c5da2-a187-4ad5-876f-35db0c631a06" containerName="maas-api"
Oct 30 01:08:07.211695 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211623    6099 cpu_manager.go:414] "containerMap: removing container" podUID="c789663e-00c2-4c17-9b12-784459a7a9e4" containerName="maas-api"
Oct 30 01:08:07.211695 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211629    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c789663e-00c2-4c17-9b12-784459a7a9e4" containerName="maas-api"
Oct 30 01:08:07.211695 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211665    6099 cpu_manager.go:414] "containerMap: removing container" podUID="15ed7bd6-34b8-49f2-af6b-453b0c7f921a" containerName="payload-processing"
Oct 30 01:08:07.211695 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211671    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15ed7bd6-34b8-49f2-af6b-453b0c7f921a" containerName="payload-processing"
Oct 30 01:08:07.212024 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211889    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="261c5da2-a187-4ad5-876f-35db0c631a06" containerName="maas-api"
Oct 30 01:08:07.212024 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211908    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c789663e-00c2-4c17-9b12-784459a7a9e4" containerName="maas-api"
Oct 30 01:08:07.212024 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211925    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="15ed7bd6-34b8-49f2-af6b-453b0c7f921a" containerName="payload-processing"
Oct 30 01:08:07.212024 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211939    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85bad1f-83ed-42b6-b88a-c1388b5480b2" containerName="istio-proxy"
Oct 30 01:08:07.212024 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211953    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85bad1f-83ed-42b6-b88a-c1388b5480b2" containerName="istio-proxy"
Oct 30 01:08:07.212024 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211965    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc93c165-35ad-4bac-9b4a-5c68e4020bfa" containerName="revoke-keys"
Oct 30 01:08:07.212024 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.211973    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9f6975-a3df-4ed1-940a-a801424eec23" containerName="payload-pre-processing"
Oct 30 01:08:07.217268 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.217248    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h"
Oct 30 01:08:07.234439 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.234411    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h"]
Oct 30 01:08:07.247748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.243028    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:08:07.247748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.243044    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:08:07.364189 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.364159    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv6bl\" (UniqueName: \"kubernetes.io/projected/1142c8f9-ffd0-4231-844c-d6236e976047-kube-api-access-rv6bl\") pod \"maas-api-revoke-sub-ba1483c0-ddg2h\" (UID: \"1142c8f9-ffd0-4231-844c-d6236e976047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h"
Oct 30 01:08:07.364528 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.364505    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1142c8f9-ffd0-4231-844c-d6236e976047-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba1483c0-ddg2h\" (UID: \"1142c8f9-ffd0-4231-844c-d6236e976047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h"
Oct 30 01:08:07.364616 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.364551    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1142c8f9-ffd0-4231-844c-d6236e976047-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba1483c0-ddg2h\" (UID: \"1142c8f9-ffd0-4231-844c-d6236e976047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h"
Oct 30 01:08:07.466241 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.466151    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1142c8f9-ffd0-4231-844c-d6236e976047-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba1483c0-ddg2h\" (UID: \"1142c8f9-ffd0-4231-844c-d6236e976047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h"
Oct 30 01:08:07.466241 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.466192    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1142c8f9-ffd0-4231-844c-d6236e976047-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba1483c0-ddg2h\" (UID: \"1142c8f9-ffd0-4231-844c-d6236e976047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h"
Oct 30 01:08:07.466241 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.466234    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv6bl\" (UniqueName: \"kubernetes.io/projected/1142c8f9-ffd0-4231-844c-d6236e976047-kube-api-access-rv6bl\") pod \"maas-api-revoke-sub-ba1483c0-ddg2h\" (UID: \"1142c8f9-ffd0-4231-844c-d6236e976047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h"
Oct 30 01:08:07.466897 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.466869    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1142c8f9-ffd0-4231-844c-d6236e976047-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba1483c0-ddg2h\" (UID: \"1142c8f9-ffd0-4231-844c-d6236e976047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h"
Oct 30 01:08:07.468675 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.468655    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1142c8f9-ffd0-4231-844c-d6236e976047-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba1483c0-ddg2h\" (UID: \"1142c8f9-ffd0-4231-844c-d6236e976047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h"
Oct 30 01:08:07.489219 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.489188    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv6bl\" (UniqueName: \"kubernetes.io/projected/1142c8f9-ffd0-4231-844c-d6236e976047-kube-api-access-rv6bl\") pod \"maas-api-revoke-sub-ba1483c0-ddg2h\" (UID: \"1142c8f9-ffd0-4231-844c-d6236e976047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h"
Oct 30 01:08:07.532491 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.532468    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h"
Oct 30 01:08:07.838346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:07.838314    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h"]
Oct 30 01:08:07.838493 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:08:07.838339    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1142c8f9_ffd0_4231_844c_d6236e976047.slice/crio-1e28b861917ff5f0469a952600f6a5db2e7db9c4cf71e7727178a819ce09d092 WatchSource:0}: Error finding container 1e28b861917ff5f0469a952600f6a5db2e7db9c4cf71e7727178a819ce09d092: Status 404 returned error can't find the container with id 1e28b861917ff5f0469a952600f6a5db2e7db9c4cf71e7727178a819ce09d092
Oct 30 01:08:08.055784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:08.055735    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h" event={"ID":"1142c8f9-ffd0-4231-844c-d6236e976047","Type":"ContainerStarted","Data":"1e28b861917ff5f0469a952600f6a5db2e7db9c4cf71e7727178a819ce09d092"}
Oct 30 01:08:09.065718 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:09.065618    6099 generic.go:356] "Generic (PLEG): container finished" podID="1142c8f9-ffd0-4231-844c-d6236e976047" containerID="055b0b5255a899c0f6ebd4b7b55e2c75ba4d602628f6ff817dd02e4a2b4cc4af" exitCode=0
Oct 30 01:08:09.066139 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:09.065730    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h" event={"ID":"1142c8f9-ffd0-4231-844c-d6236e976047","Type":"ContainerDied","Data":"055b0b5255a899c0f6ebd4b7b55e2c75ba4d602628f6ff817dd02e4a2b4cc4af"}
Oct 30 01:08:09.921242 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:09.921205    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd" podUID="66824435-d248-47f0-ab98-f8f2c53e25b1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:08:10.242842 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.242812    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h"
Oct 30 01:08:10.414457 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.414423    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1142c8f9-ffd0-4231-844c-d6236e976047-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1142c8f9-ffd0-4231-844c-d6236e976047-maas-api-xks-ca\") pod \"1142c8f9-ffd0-4231-844c-d6236e976047\" (UID: \"1142c8f9-ffd0-4231-844c-d6236e976047\") "
Oct 30 01:08:10.414647 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.414589    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1142c8f9-ffd0-4231-844c-d6236e976047-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1142c8f9-ffd0-4231-844c-d6236e976047-maas-api-openshift-ca\") pod \"1142c8f9-ffd0-4231-844c-d6236e976047\" (UID: \"1142c8f9-ffd0-4231-844c-d6236e976047\") "
Oct 30 01:08:10.414753 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.414697    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1142c8f9-ffd0-4231-844c-d6236e976047-kube-api-access-rv6bl\" (UniqueName: \"kubernetes.io/projected/1142c8f9-ffd0-4231-844c-d6236e976047-kube-api-access-rv6bl\") pod \"1142c8f9-ffd0-4231-844c-d6236e976047\" (UID: \"1142c8f9-ffd0-4231-844c-d6236e976047\") "
Oct 30 01:08:10.414954 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.414928    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1142c8f9-ffd0-4231-844c-d6236e976047-maas-api-openshift-ca" pod "1142c8f9-ffd0-4231-844c-d6236e976047" (UID: "1142c8f9-ffd0-4231-844c-d6236e976047"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:10.415563 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.415541    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1142c8f9-ffd0-4231-844c-d6236e976047-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:10.416652 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.416631    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1142c8f9-ffd0-4231-844c-d6236e976047-maas-api-xks-ca" pod "1142c8f9-ffd0-4231-844c-d6236e976047" (UID: "1142c8f9-ffd0-4231-844c-d6236e976047"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:10.416791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.416772    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1142c8f9-ffd0-4231-844c-d6236e976047-kube-api-access-rv6bl" pod "1142c8f9-ffd0-4231-844c-d6236e976047" (UID: "1142c8f9-ffd0-4231-844c-d6236e976047"). InnerVolumeSpecName "kube-api-access-rv6bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:10.516822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.516742    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv6bl\" (UniqueName: \"kubernetes.io/projected/1142c8f9-ffd0-4231-844c-d6236e976047-kube-api-access-rv6bl\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:10.516822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.516771    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1142c8f9-ffd0-4231-844c-d6236e976047-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:10.657778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.657746    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p"]
Oct 30 01:08:10.658566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.658535    6099 cpu_manager.go:414] "containerMap: removing container" podUID="f85bad1f-83ed-42b6-b88a-c1388b5480b2" containerName="istio-proxy"
Oct 30 01:08:10.658566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.658566    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85bad1f-83ed-42b6-b88a-c1388b5480b2" containerName="istio-proxy"
Oct 30 01:08:10.658688 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.658624    6099 cpu_manager.go:414] "containerMap: removing container" podUID="1142c8f9-ffd0-4231-844c-d6236e976047" containerName="revoke-keys"
Oct 30 01:08:10.658688 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.658630    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1142c8f9-ffd0-4231-844c-d6236e976047" containerName="revoke-keys"
Oct 30 01:08:10.658788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.658768    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1142c8f9-ffd0-4231-844c-d6236e976047" containerName="revoke-keys"
Oct 30 01:08:10.662872 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.662851    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p"
Oct 30 01:08:10.682479 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.682454    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p"]
Oct 30 01:08:10.823549 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.823507    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b105e1-3566-447d-bfc8-f250043b8341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b0f2044-54t2p\" (UID: \"31b105e1-3566-447d-bfc8-f250043b8341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p"
Oct 30 01:08:10.823749 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.823565    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zcrc\" (UniqueName: \"kubernetes.io/projected/31b105e1-3566-447d-bfc8-f250043b8341-kube-api-access-5zcrc\") pod \"maas-api-revoke-sub-6b0f2044-54t2p\" (UID: \"31b105e1-3566-447d-bfc8-f250043b8341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p"
Oct 30 01:08:10.823856 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.823842    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b105e1-3566-447d-bfc8-f250043b8341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b0f2044-54t2p\" (UID: \"31b105e1-3566-447d-bfc8-f250043b8341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p"
Oct 30 01:08:10.924785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.924740    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b105e1-3566-447d-bfc8-f250043b8341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b0f2044-54t2p\" (UID: \"31b105e1-3566-447d-bfc8-f250043b8341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p"
Oct 30 01:08:10.924990 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.924803    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zcrc\" (UniqueName: \"kubernetes.io/projected/31b105e1-3566-447d-bfc8-f250043b8341-kube-api-access-5zcrc\") pod \"maas-api-revoke-sub-6b0f2044-54t2p\" (UID: \"31b105e1-3566-447d-bfc8-f250043b8341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p"
Oct 30 01:08:10.924990 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.924976    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b105e1-3566-447d-bfc8-f250043b8341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b0f2044-54t2p\" (UID: \"31b105e1-3566-447d-bfc8-f250043b8341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p"
Oct 30 01:08:10.925459 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.925433    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b105e1-3566-447d-bfc8-f250043b8341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b0f2044-54t2p\" (UID: \"31b105e1-3566-447d-bfc8-f250043b8341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p"
Oct 30 01:08:10.927348 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.927327    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b105e1-3566-447d-bfc8-f250043b8341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b0f2044-54t2p\" (UID: \"31b105e1-3566-447d-bfc8-f250043b8341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p"
Oct 30 01:08:10.947574 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.947546    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zcrc\" (UniqueName: \"kubernetes.io/projected/31b105e1-3566-447d-bfc8-f250043b8341-kube-api-access-5zcrc\") pod \"maas-api-revoke-sub-6b0f2044-54t2p\" (UID: \"31b105e1-3566-447d-bfc8-f250043b8341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p"
Oct 30 01:08:10.977623 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:10.977600    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p"
Oct 30 01:08:11.087321 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:11.087245    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h"
Oct 30 01:08:11.087491 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:11.087373    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h" event={"ID":"1142c8f9-ffd0-4231-844c-d6236e976047","Type":"ContainerDied","Data":"1e28b861917ff5f0469a952600f6a5db2e7db9c4cf71e7727178a819ce09d092"}
Oct 30 01:08:11.087491 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:11.087437    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e28b861917ff5f0469a952600f6a5db2e7db9c4cf71e7727178a819ce09d092"
Oct 30 01:08:11.284720 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:11.284673    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p"]
Oct 30 01:08:11.285064 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:08:11.284979    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b105e1_3566_447d_bfc8_f250043b8341.slice/crio-2b87d17fe33ff65838148530da248af3672470b0f8755052258d2a4f3c716bad WatchSource:0}: Error finding container 2b87d17fe33ff65838148530da248af3672470b0f8755052258d2a4f3c716bad: Status 404 returned error can't find the container with id 2b87d17fe33ff65838148530da248af3672470b0f8755052258d2a4f3c716bad
Oct 30 01:08:12.144729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:12.141780    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p" event={"ID":"31b105e1-3566-447d-bfc8-f250043b8341","Type":"ContainerStarted","Data":"450dcd6b2434b2ebe33281cbfab2e1560406f1c73079a4f859015e6f9aad7971"}
Oct 30 01:08:12.144729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:12.141849    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p" event={"ID":"31b105e1-3566-447d-bfc8-f250043b8341","Type":"ContainerStarted","Data":"2b87d17fe33ff65838148530da248af3672470b0f8755052258d2a4f3c716bad"}
Oct 30 01:08:12.252295 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:12.252244    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p" podStartSLOduration=2.252226306 podStartE2EDuration="2.252226306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:12.251780879 +0000 UTC m=+1986.108688637" watchObservedRunningTime="2026-10-30 01:08:12.252226306 +0000 UTC m=+1986.109134060"
Oct 30 01:08:13.153183 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:13.153149    6099 generic.go:356] "Generic (PLEG): container finished" podID="31b105e1-3566-447d-bfc8-f250043b8341" containerID="450dcd6b2434b2ebe33281cbfab2e1560406f1c73079a4f859015e6f9aad7971" exitCode=0
Oct 30 01:08:13.153551 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:13.153237    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p" event={"ID":"31b105e1-3566-447d-bfc8-f250043b8341","Type":"ContainerDied","Data":"450dcd6b2434b2ebe33281cbfab2e1560406f1c73079a4f859015e6f9aad7971"}
Oct 30 01:08:13.794973 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:13.794936    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"]
Oct 30 01:08:13.795216 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:13.795191    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf" podUID="c9a34143-9d52-4d6a-99d3-05d6501f1bb7" containerName="istio-proxy" containerID="cri-o://a20a492200614723d9316ac3ac8826dc9488350f5a7c6baaa3db7da36b97f122" gracePeriod=30
Oct 30 01:08:14.342208 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:14.342181    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p"
Oct 30 01:08:14.410866 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:14.410839    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b105e1-3566-447d-bfc8-f250043b8341-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b105e1-3566-447d-bfc8-f250043b8341-maas-api-openshift-ca\") pod \"31b105e1-3566-447d-bfc8-f250043b8341\" (UID: \"31b105e1-3566-447d-bfc8-f250043b8341\") "
Oct 30 01:08:14.411036 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:14.410892    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31b105e1-3566-447d-bfc8-f250043b8341-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b105e1-3566-447d-bfc8-f250043b8341-maas-api-xks-ca\") pod \"31b105e1-3566-447d-bfc8-f250043b8341\" (UID: \"31b105e1-3566-447d-bfc8-f250043b8341\") "
Oct 30 01:08:14.411090 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:14.411059    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b105e1-3566-447d-bfc8-f250043b8341-kube-api-access-5zcrc\" (UniqueName: \"kubernetes.io/projected/31b105e1-3566-447d-bfc8-f250043b8341-kube-api-access-5zcrc\") pod \"31b105e1-3566-447d-bfc8-f250043b8341\" (UID: \"31b105e1-3566-447d-bfc8-f250043b8341\") "
Oct 30 01:08:14.411316 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:14.411286    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b105e1-3566-447d-bfc8-f250043b8341-maas-api-openshift-ca" pod "31b105e1-3566-447d-bfc8-f250043b8341" (UID: "31b105e1-3566-447d-bfc8-f250043b8341"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:14.411987 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:14.411928    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b105e1-3566-447d-bfc8-f250043b8341-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:14.413064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:14.413039    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31b105e1-3566-447d-bfc8-f250043b8341-maas-api-xks-ca" pod "31b105e1-3566-447d-bfc8-f250043b8341" (UID: "31b105e1-3566-447d-bfc8-f250043b8341"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:14.413192 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:14.413172    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b105e1-3566-447d-bfc8-f250043b8341-kube-api-access-5zcrc" pod "31b105e1-3566-447d-bfc8-f250043b8341" (UID: "31b105e1-3566-447d-bfc8-f250043b8341"). InnerVolumeSpecName "kube-api-access-5zcrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:14.514425 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:14.514397    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b105e1-3566-447d-bfc8-f250043b8341-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:14.514425 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:14.514426    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zcrc\" (UniqueName: \"kubernetes.io/projected/31b105e1-3566-447d-bfc8-f250043b8341-kube-api-access-5zcrc\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:15.048546 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.048510    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"]
Oct 30 01:08:15.049666 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.049637    6099 cpu_manager.go:414] "containerMap: removing container" podUID="31b105e1-3566-447d-bfc8-f250043b8341" containerName="revoke-keys"
Oct 30 01:08:15.049666 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.049665    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b105e1-3566-447d-bfc8-f250043b8341" containerName="revoke-keys"
Oct 30 01:08:15.049885 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.049873    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b105e1-3566-447d-bfc8-f250043b8341" containerName="revoke-keys"
Oct 30 01:08:15.053779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.053760    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.070314 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.070288    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-6c4f6f-openshift-default-dockercfg-gp89t\""
Oct 30 01:08:15.098839 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.098806    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"]
Oct 30 01:08:15.125299 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.125263    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-envoy\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.125460 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.125431    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-workload-socket\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.126005 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.125480    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-workload-certs\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.126005 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.125534    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-token\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.126005 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.125573    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-podinfo\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.126005 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.125737    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v29s\" (UniqueName: \"kubernetes.io/projected/caf7a449-3657-4b2e-aaa6-2cca317f241f-kube-api-access-5v29s\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.126005 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.125850    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-credential-socket\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.126005 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.125987    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf7a449-3657-4b2e-aaa6-2cca317f241f-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.126369 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.126033    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-data\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.187736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.184599    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p"
Oct 30 01:08:15.187736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.184601    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p" event={"ID":"31b105e1-3566-447d-bfc8-f250043b8341","Type":"ContainerDied","Data":"2b87d17fe33ff65838148530da248af3672470b0f8755052258d2a4f3c716bad"}
Oct 30 01:08:15.187736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.184753    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b87d17fe33ff65838148530da248af3672470b0f8755052258d2a4f3c716bad"
Oct 30 01:08:15.251728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.248936    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-credential-socket\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.251728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.249095    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf7a449-3657-4b2e-aaa6-2cca317f241f-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.251728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.249162    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-data\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.251728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.249232    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-envoy\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.251728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.249388    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-workload-socket\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.251728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.249434    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-workload-certs\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.251728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.249461    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-token\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.251728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.249500    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-podinfo\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.251728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.249557    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v29s\" (UniqueName: \"kubernetes.io/projected/caf7a449-3657-4b2e-aaa6-2cca317f241f-kube-api-access-5v29s\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.251728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.251272    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf7a449-3657-4b2e-aaa6-2cca317f241f-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.280750 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.259058    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-workload-socket\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.280750 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.260812    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-credential-socket\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.280750 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.261109    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-workload-certs\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.280750 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.261331    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-data\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.303770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.284053    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-envoy\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.303770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.286837    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-podinfo\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.303770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.287568    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v29s\" (UniqueName: \"kubernetes.io/projected/caf7a449-3657-4b2e-aaa6-2cca317f241f-kube-api-access-5v29s\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.303770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.287943    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-token\") pod \"e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") " pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.374732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.374043    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:15.684394 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.684368    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd"
Oct 30 01:08:15.709221 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.709181    6099 patch_prober.go:29] interesting pod/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:08:15.709389 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.709242    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf" podUID="c9a34143-9d52-4d6a-99d3-05d6501f1bb7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:08:15.764805 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.764766    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66824435-d248-47f0-ab98-f8f2c53e25b1-kube-api-access-hk8fh\" (UniqueName: \"kubernetes.io/projected/66824435-d248-47f0-ab98-f8f2c53e25b1-kube-api-access-hk8fh\") pod \"66824435-d248-47f0-ab98-f8f2c53e25b1\" (UID: \"66824435-d248-47f0-ab98-f8f2c53e25b1\") "
Oct 30 01:08:15.764978 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.764852    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66824435-d248-47f0-ab98-f8f2c53e25b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66824435-d248-47f0-ab98-f8f2c53e25b1-plugins-config-volume\") pod \"66824435-d248-47f0-ab98-f8f2c53e25b1\" (UID: \"66824435-d248-47f0-ab98-f8f2c53e25b1\") "
Oct 30 01:08:15.765334 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.765305    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66824435-d248-47f0-ab98-f8f2c53e25b1-plugins-config-volume" pod "66824435-d248-47f0-ab98-f8f2c53e25b1" (UID: "66824435-d248-47f0-ab98-f8f2c53e25b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:15.765944 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.765922    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66824435-d248-47f0-ab98-f8f2c53e25b1-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:15.767144 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.767115    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66824435-d248-47f0-ab98-f8f2c53e25b1-kube-api-access-hk8fh" pod "66824435-d248-47f0-ab98-f8f2c53e25b1" (UID: "66824435-d248-47f0-ab98-f8f2c53e25b1"). InnerVolumeSpecName "kube-api-access-hk8fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:15.838648 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:08:15.838617    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf7a449_3657_4b2e_aaa6_2cca317f241f.slice/crio-cd07493f4492ef89e007834f1e3165139306bc5ff1f1e0e5b28be6b6ab32c95f WatchSource:0}: Error finding container cd07493f4492ef89e007834f1e3165139306bc5ff1f1e0e5b28be6b6ab32c95f: Status 404 returned error can't find the container with id cd07493f4492ef89e007834f1e3165139306bc5ff1f1e0e5b28be6b6ab32c95f
Oct 30 01:08:15.839441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.839418    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"]
Oct 30 01:08:15.841628 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.841591    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:08:15.841769 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.841668    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:08:15.841841 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.841768    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:08:15.867458 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:15.867434    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk8fh\" (UniqueName: \"kubernetes.io/projected/66824435-d248-47f0-ab98-f8f2c53e25b1-kube-api-access-hk8fh\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:16.198012 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:16.197918    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2" event={"ID":"caf7a449-3657-4b2e-aaa6-2cca317f241f","Type":"ContainerStarted","Data":"c35efa95ab8313096506a40443269c2490bad57fae4fa5d348ce8f09e728f8b9"}
Oct 30 01:08:16.198012 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:16.197980    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2" event={"ID":"caf7a449-3657-4b2e-aaa6-2cca317f241f","Type":"ContainerStarted","Data":"cd07493f4492ef89e007834f1e3165139306bc5ff1f1e0e5b28be6b6ab32c95f"}
Oct 30 01:08:16.199463 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:16.199429    6099 generic.go:356] "Generic (PLEG): container finished" podID="66824435-d248-47f0-ab98-f8f2c53e25b1" containerID="c1198035f1cbb5fa1a4e781a9437167344d573b84a79bc5913e84c011c21e342" exitCode=0
Oct 30 01:08:16.199589 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:16.199462    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd"
Oct 30 01:08:16.199589 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:16.199500    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd" event={"ID":"66824435-d248-47f0-ab98-f8f2c53e25b1","Type":"ContainerDied","Data":"c1198035f1cbb5fa1a4e781a9437167344d573b84a79bc5913e84c011c21e342"}
Oct 30 01:08:16.199589 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:16.199546    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd" event={"ID":"66824435-d248-47f0-ab98-f8f2c53e25b1","Type":"ContainerDied","Data":"b747ad293b17e92fd699e5553057b93e65c0b85e53f8aced9117b8d3c07ce767"}
Oct 30 01:08:16.199589 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:16.199569    6099 scope.go:122] "RemoveContainer" containerID="c1198035f1cbb5fa1a4e781a9437167344d573b84a79bc5913e84c011c21e342"
Oct 30 01:08:16.211555 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:16.211531    6099 scope.go:122] "RemoveContainer" containerID="c1198035f1cbb5fa1a4e781a9437167344d573b84a79bc5913e84c011c21e342"
Oct 30 01:08:16.211897 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:08:16.211870    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1198035f1cbb5fa1a4e781a9437167344d573b84a79bc5913e84c011c21e342\": container with ID starting with c1198035f1cbb5fa1a4e781a9437167344d573b84a79bc5913e84c011c21e342 not found: ID does not exist" containerID="c1198035f1cbb5fa1a4e781a9437167344d573b84a79bc5913e84c011c21e342"
Oct 30 01:08:16.211988 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:16.211907    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1198035f1cbb5fa1a4e781a9437167344d573b84a79bc5913e84c011c21e342"} err="failed to get container status \"c1198035f1cbb5fa1a4e781a9437167344d573b84a79bc5913e84c011c21e342\": rpc error: code = NotFound desc = could not find container \"c1198035f1cbb5fa1a4e781a9437167344d573b84a79bc5913e84c011c21e342\": container with ID starting with c1198035f1cbb5fa1a4e781a9437167344d573b84a79bc5913e84c011c21e342 not found: ID does not exist"
Oct 30 01:08:16.247347 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:16.247296    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2" podStartSLOduration=1.247282362 podStartE2EDuration="1.247282362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:16.241832543 +0000 UTC m=+1990.098740298" watchObservedRunningTime="2026-10-30 01:08:16.247282362 +0000 UTC m=+1990.104190178"
Oct 30 01:08:16.283511 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:16.283480    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd"]
Oct 30 01:08:16.290338 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:16.290309    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-820c04-5cd68568bc-q44kd"]
Oct 30 01:08:16.393728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:16.387050    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:16.406048 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:16.406009    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:16.631254 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:08:16.631219    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:08:17.208495 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:17.208466    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:17.209509 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:17.209489    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:08:18.262967 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:18.262898    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66824435-d248-47f0-ab98-f8f2c53e25b1" path="/var/lib/kubelet/pods/66824435-d248-47f0-ab98-f8f2c53e25b1/volumes"
Oct 30 01:08:19.215942 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.215917    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:08:19.224434 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.224406    6099 generic.go:356] "Generic (PLEG): container finished" podID="c9a34143-9d52-4d6a-99d3-05d6501f1bb7" containerID="a20a492200614723d9316ac3ac8826dc9488350f5a7c6baaa3db7da36b97f122" exitCode=0
Oct 30 01:08:19.224596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.224455    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"
Oct 30 01:08:19.224596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.224493    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf" event={"ID":"c9a34143-9d52-4d6a-99d3-05d6501f1bb7","Type":"ContainerDied","Data":"a20a492200614723d9316ac3ac8826dc9488350f5a7c6baaa3db7da36b97f122"}
Oct 30 01:08:19.224596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.224561    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf" event={"ID":"c9a34143-9d52-4d6a-99d3-05d6501f1bb7","Type":"ContainerDied","Data":"d326bfff99249bcd76becca7d06ef0fa6abe01543c850be4bc758995a8fa298e"}
Oct 30 01:08:19.224596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.224586    6099 scope.go:122] "RemoveContainer" containerID="a20a492200614723d9316ac3ac8826dc9488350f5a7c6baaa3db7da36b97f122"
Oct 30 01:08:19.235359 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.235340    6099 scope.go:122] "RemoveContainer" containerID="86971e1ba50bbb98ffa4e637644d94ed90d0edbc81b42f053f4584241f82a27a"
Oct 30 01:08:19.244921 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.244905    6099 scope.go:122] "RemoveContainer" containerID="a20a492200614723d9316ac3ac8826dc9488350f5a7c6baaa3db7da36b97f122"
Oct 30 01:08:19.245175 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:08:19.245157    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a20a492200614723d9316ac3ac8826dc9488350f5a7c6baaa3db7da36b97f122\": container with ID starting with a20a492200614723d9316ac3ac8826dc9488350f5a7c6baaa3db7da36b97f122 not found: ID does not exist" containerID="a20a492200614723d9316ac3ac8826dc9488350f5a7c6baaa3db7da36b97f122"
Oct 30 01:08:19.245230 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.245183    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20a492200614723d9316ac3ac8826dc9488350f5a7c6baaa3db7da36b97f122"} err="failed to get container status \"a20a492200614723d9316ac3ac8826dc9488350f5a7c6baaa3db7da36b97f122\": rpc error: code = NotFound desc = could not find container \"a20a492200614723d9316ac3ac8826dc9488350f5a7c6baaa3db7da36b97f122\": container with ID starting with a20a492200614723d9316ac3ac8826dc9488350f5a7c6baaa3db7da36b97f122 not found: ID does not exist"
Oct 30 01:08:19.245230 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.245207    6099 scope.go:122] "RemoveContainer" containerID="86971e1ba50bbb98ffa4e637644d94ed90d0edbc81b42f053f4584241f82a27a"
Oct 30 01:08:19.245504 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:08:19.245486    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86971e1ba50bbb98ffa4e637644d94ed90d0edbc81b42f053f4584241f82a27a\": container with ID starting with 86971e1ba50bbb98ffa4e637644d94ed90d0edbc81b42f053f4584241f82a27a not found: ID does not exist" containerID="86971e1ba50bbb98ffa4e637644d94ed90d0edbc81b42f053f4584241f82a27a"
Oct 30 01:08:19.245554 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.245510    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86971e1ba50bbb98ffa4e637644d94ed90d0edbc81b42f053f4584241f82a27a"} err="failed to get container status \"86971e1ba50bbb98ffa4e637644d94ed90d0edbc81b42f053f4584241f82a27a\": rpc error: code = NotFound desc = could not find container \"86971e1ba50bbb98ffa4e637644d94ed90d0edbc81b42f053f4584241f82a27a\": container with ID starting with 86971e1ba50bbb98ffa4e637644d94ed90d0edbc81b42f053f4584241f82a27a not found: ID does not exist"
Oct 30 01:08:19.343138 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.343096    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-workload-socket\") pod \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") "
Oct 30 01:08:19.343579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.343148    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-envoy\") pod \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") "
Oct 30 01:08:19.343579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.343298    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-workload-certs\") pod \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") "
Oct 30 01:08:19.343579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.343423    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istiod-ca-cert\") pod \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") "
Oct 30 01:08:19.343579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.343437    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-workload-socket" pod "c9a34143-9d52-4d6a-99d3-05d6501f1bb7" (UID: "c9a34143-9d52-4d6a-99d3-05d6501f1bb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:19.343579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.343479    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-token\" (UniqueName: \"kubernetes.io/projected/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-token\") pod \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") "
Oct 30 01:08:19.343579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.343524    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-credential-socket\") pod \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") "
Oct 30 01:08:19.343970 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.343590    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-data\") pod \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") "
Oct 30 01:08:19.343970 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.343624    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-podinfo\") pod \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") "
Oct 30 01:08:19.343970 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.343654    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-kube-api-access-2x7x6\" (UniqueName: \"kubernetes.io/projected/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-kube-api-access-2x7x6\") pod \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\" (UID: \"c9a34143-9d52-4d6a-99d3-05d6501f1bb7\") "
Oct 30 01:08:19.343970 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.343804    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istiod-ca-cert" pod "c9a34143-9d52-4d6a-99d3-05d6501f1bb7" (UID: "c9a34143-9d52-4d6a-99d3-05d6501f1bb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:19.343970 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.343914    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-workload-certs" pod "c9a34143-9d52-4d6a-99d3-05d6501f1bb7" (UID: "c9a34143-9d52-4d6a-99d3-05d6501f1bb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:19.344251 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.344070    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-data" pod "c9a34143-9d52-4d6a-99d3-05d6501f1bb7" (UID: "c9a34143-9d52-4d6a-99d3-05d6501f1bb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:19.344251 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.344157    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-credential-socket" pod "c9a34143-9d52-4d6a-99d3-05d6501f1bb7" (UID: "c9a34143-9d52-4d6a-99d3-05d6501f1bb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:19.344873 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.344846    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:19.344873 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.344876    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:19.345081 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.344890    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:19.345081 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.344904    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:19.345081 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.344917    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:19.346404 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.346355    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-envoy" pod "c9a34143-9d52-4d6a-99d3-05d6501f1bb7" (UID: "c9a34143-9d52-4d6a-99d3-05d6501f1bb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:19.346858 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.346831    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-token" pod "c9a34143-9d52-4d6a-99d3-05d6501f1bb7" (UID: "c9a34143-9d52-4d6a-99d3-05d6501f1bb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:19.347744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.347687    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-podinfo" pod "c9a34143-9d52-4d6a-99d3-05d6501f1bb7" (UID: "c9a34143-9d52-4d6a-99d3-05d6501f1bb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:08:19.348291 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.348267    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-kube-api-access-2x7x6" pod "c9a34143-9d52-4d6a-99d3-05d6501f1bb7" (UID: "c9a34143-9d52-4d6a-99d3-05d6501f1bb7"). InnerVolumeSpecName "kube-api-access-2x7x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:19.446490 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.446421    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:19.446490 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.446447    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x7x6\" (UniqueName: \"kubernetes.io/projected/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-kube-api-access-2x7x6\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:19.446490 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.446458    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:19.446490 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.446466    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9a34143-9d52-4d6a-99d3-05d6501f1bb7-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:19.581579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.581550    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"]
Oct 30 01:08:19.587576 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:19.587548    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8b26-openshift-default-84b87df95f-j98mf"]
Oct 30 01:08:20.263440 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.263411    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9a34143-9d52-4d6a-99d3-05d6501f1bb7" path="/var/lib/kubelet/pods/c9a34143-9d52-4d6a-99d3-05d6501f1bb7/volumes"
Oct 30 01:08:20.512824 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.512787    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s"]
Oct 30 01:08:20.513631 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.513584    6099 cpu_manager.go:414] "containerMap: removing container" podUID="66824435-d248-47f0-ab98-f8f2c53e25b1" containerName="payload-pre-processing"
Oct 30 01:08:20.513631 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.513604    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66824435-d248-47f0-ab98-f8f2c53e25b1" containerName="payload-pre-processing"
Oct 30 01:08:20.513631 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.513620    6099 cpu_manager.go:414] "containerMap: removing container" podUID="c9a34143-9d52-4d6a-99d3-05d6501f1bb7" containerName="istio-proxy"
Oct 30 01:08:20.513631 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.513625    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a34143-9d52-4d6a-99d3-05d6501f1bb7" containerName="istio-proxy"
Oct 30 01:08:20.513904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.513639    6099 cpu_manager.go:414] "containerMap: removing container" podUID="c9a34143-9d52-4d6a-99d3-05d6501f1bb7" containerName="istio-proxy"
Oct 30 01:08:20.513904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.513644    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a34143-9d52-4d6a-99d3-05d6501f1bb7" containerName="istio-proxy"
Oct 30 01:08:20.513904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.513840    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="66824435-d248-47f0-ab98-f8f2c53e25b1" containerName="payload-pre-processing"
Oct 30 01:08:20.513904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.513851    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9a34143-9d52-4d6a-99d3-05d6501f1bb7" containerName="istio-proxy"
Oct 30 01:08:20.513904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.513859    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9a34143-9d52-4d6a-99d3-05d6501f1bb7" containerName="istio-proxy"
Oct 30 01:08:20.519627 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.519597    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s"
Oct 30 01:08:20.538101 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.538073    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s"]
Oct 30 01:08:20.543469 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.543444    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-6c4f6f-dockercfg-kq7gx\""
Oct 30 01:08:20.543688 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.543525    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-6c4f6f\""
Oct 30 01:08:20.556378 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.556352    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx"]
Oct 30 01:08:20.563312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.563291    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx"
Oct 30 01:08:20.573910 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.573487    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx"]
Oct 30 01:08:20.699809 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.699536    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a003430b-13be-47b6-864c-9062ae70f676-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx\" (UID: \"a003430b-13be-47b6-864c-9062ae70f676\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx"
Oct 30 01:08:20.700035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.699855    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbdkj\" (UniqueName: \"kubernetes.io/projected/5eb2c17f-f68a-4672-b266-f3562154baa3-kube-api-access-lbdkj\") pod \"payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s\" (UID: \"5eb2c17f-f68a-4672-b266-f3562154baa3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s"
Oct 30 01:08:20.700035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.699918    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb2c17f-f68a-4672-b266-f3562154baa3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s\" (UID: \"5eb2c17f-f68a-4672-b266-f3562154baa3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s"
Oct 30 01:08:20.700178 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.700097    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmf4b\" (UniqueName: \"kubernetes.io/projected/a003430b-13be-47b6-864c-9062ae70f676-kube-api-access-nmf4b\") pod \"payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx\" (UID: \"a003430b-13be-47b6-864c-9062ae70f676\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx"
Oct 30 01:08:20.853434 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.851142    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbdkj\" (UniqueName: \"kubernetes.io/projected/5eb2c17f-f68a-4672-b266-f3562154baa3-kube-api-access-lbdkj\") pod \"payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s\" (UID: \"5eb2c17f-f68a-4672-b266-f3562154baa3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s"
Oct 30 01:08:20.853434 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.851247    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb2c17f-f68a-4672-b266-f3562154baa3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s\" (UID: \"5eb2c17f-f68a-4672-b266-f3562154baa3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s"
Oct 30 01:08:20.853434 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.851425    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmf4b\" (UniqueName: \"kubernetes.io/projected/a003430b-13be-47b6-864c-9062ae70f676-kube-api-access-nmf4b\") pod \"payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx\" (UID: \"a003430b-13be-47b6-864c-9062ae70f676\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx"
Oct 30 01:08:20.853434 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.851498    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a003430b-13be-47b6-864c-9062ae70f676-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx\" (UID: \"a003430b-13be-47b6-864c-9062ae70f676\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx"
Oct 30 01:08:20.853434 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.852761    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a003430b-13be-47b6-864c-9062ae70f676-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx\" (UID: \"a003430b-13be-47b6-864c-9062ae70f676\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx"
Oct 30 01:08:20.885636 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.885591    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb2c17f-f68a-4672-b266-f3562154baa3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s\" (UID: \"5eb2c17f-f68a-4672-b266-f3562154baa3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s"
Oct 30 01:08:20.905774 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.901871    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmf4b\" (UniqueName: \"kubernetes.io/projected/a003430b-13be-47b6-864c-9062ae70f676-kube-api-access-nmf4b\") pod \"payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx\" (UID: \"a003430b-13be-47b6-864c-9062ae70f676\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx"
Oct 30 01:08:20.905774 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:20.902362    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbdkj\" (UniqueName: \"kubernetes.io/projected/5eb2c17f-f68a-4672-b266-f3562154baa3-kube-api-access-lbdkj\") pod \"payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s\" (UID: \"5eb2c17f-f68a-4672-b266-f3562154baa3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s"
Oct 30 01:08:21.141053 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:21.140965    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s"
Oct 30 01:08:21.181491 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:21.181461    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx"
Oct 30 01:08:21.451120 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:08:21.451089    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb2c17f_f68a_4672_b266_f3562154baa3.slice/crio-d01a035737fb1873d3275efe7389f95ee54dc6bd53ca2b04c4686f70b57c6c66 WatchSource:0}: Error finding container d01a035737fb1873d3275efe7389f95ee54dc6bd53ca2b04c4686f70b57c6c66: Status 404 returned error can't find the container with id d01a035737fb1873d3275efe7389f95ee54dc6bd53ca2b04c4686f70b57c6c66
Oct 30 01:08:21.451798 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:21.451770    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s"]
Oct 30 01:08:21.538006 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:08:21.530840    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda003430b_13be_47b6_864c_9062ae70f676.slice/crio-f9af8eae4a71cbd8d60fa34fca88ce3286bc12d180010a772f3b11d676812bb4 WatchSource:0}: Error finding container f9af8eae4a71cbd8d60fa34fca88ce3286bc12d180010a772f3b11d676812bb4: Status 404 returned error can't find the container with id f9af8eae4a71cbd8d60fa34fca88ce3286bc12d180010a772f3b11d676812bb4
Oct 30 01:08:21.538006 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:21.535439    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx"]
Oct 30 01:08:22.273524 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:22.273488    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s" event={"ID":"5eb2c17f-f68a-4672-b266-f3562154baa3","Type":"ContainerStarted","Data":"347b93dfefbcdc70315113b5e4bc91862a2a1406f877c10398a06a42761a823f"}
Oct 30 01:08:22.273780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:22.273544    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s" event={"ID":"5eb2c17f-f68a-4672-b266-f3562154baa3","Type":"ContainerStarted","Data":"d01a035737fb1873d3275efe7389f95ee54dc6bd53ca2b04c4686f70b57c6c66"}
Oct 30 01:08:22.273780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:22.273726    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s"
Oct 30 01:08:22.275163 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:22.275138    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx" event={"ID":"a003430b-13be-47b6-864c-9062ae70f676","Type":"ContainerStarted","Data":"307aba3e165a282d1d9ec24ecb2f268308dea98948b0cf9e8058c95792938335"}
Oct 30 01:08:22.275271 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:22.275180    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx" event={"ID":"a003430b-13be-47b6-864c-9062ae70f676","Type":"ContainerStarted","Data":"f9af8eae4a71cbd8d60fa34fca88ce3286bc12d180010a772f3b11d676812bb4"}
Oct 30 01:08:22.275271 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:22.275228    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx"
Oct 30 01:08:22.319253 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:22.319203    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s" podStartSLOduration=2.319188887 podStartE2EDuration="2.319188887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:22.31760629 +0000 UTC m=+1996.174514044" watchObservedRunningTime="2026-10-30 01:08:22.319188887 +0000 UTC m=+1996.176096637"
Oct 30 01:08:22.359590 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:22.359537    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx" podStartSLOduration=2.3595244 podStartE2EDuration="2.3595244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:22.356673218 +0000 UTC m=+1996.213580972" watchObservedRunningTime="2026-10-30 01:08:22.3595244 +0000 UTC m=+1996.216432155"
Oct 30 01:08:23.045716 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:23.045670    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd"]
Oct 30 01:08:23.051465 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:23.051438    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd"
Oct 30 01:08:23.069298 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:23.068608    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:08:23.073459 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:23.073427    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd"]
Oct 30 01:08:23.076537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:23.076515    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:08:23.199880 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:23.199846    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ffb27d0-gpjqd\" (UID: \"7aa50df9-7fa7-48ac-a05b-3cd783aee9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd"
Oct 30 01:08:23.200073 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:23.200057    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr8sh\" (UniqueName: \"kubernetes.io/projected/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-kube-api-access-zr8sh\") pod \"maas-api-revoke-sub-1ffb27d0-gpjqd\" (UID: \"7aa50df9-7fa7-48ac-a05b-3cd783aee9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd"
Oct 30 01:08:23.200185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:23.200170    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ffb27d0-gpjqd\" (UID: \"7aa50df9-7fa7-48ac-a05b-3cd783aee9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd"
Oct 30 01:08:23.303799 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:23.302844    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ffb27d0-gpjqd\" (UID: \"7aa50df9-7fa7-48ac-a05b-3cd783aee9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd"
Oct 30 01:08:23.311919 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:23.306294    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr8sh\" (UniqueName: \"kubernetes.io/projected/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-kube-api-access-zr8sh\") pod \"maas-api-revoke-sub-1ffb27d0-gpjqd\" (UID: \"7aa50df9-7fa7-48ac-a05b-3cd783aee9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd"
Oct 30 01:08:23.311919 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:23.306829    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ffb27d0-gpjqd\" (UID: \"7aa50df9-7fa7-48ac-a05b-3cd783aee9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd"
Oct 30 01:08:23.315921 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:23.312283    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ffb27d0-gpjqd\" (UID: \"7aa50df9-7fa7-48ac-a05b-3cd783aee9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd"
Oct 30 01:08:23.316124 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:23.316100    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ffb27d0-gpjqd\" (UID: \"7aa50df9-7fa7-48ac-a05b-3cd783aee9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd"
Oct 30 01:08:23.335867 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:23.335834    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr8sh\" (UniqueName: \"kubernetes.io/projected/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-kube-api-access-zr8sh\") pod \"maas-api-revoke-sub-1ffb27d0-gpjqd\" (UID: \"7aa50df9-7fa7-48ac-a05b-3cd783aee9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd"
Oct 30 01:08:23.378009 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:23.377981    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd"
Oct 30 01:08:23.686352 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:08:23.686316    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa50df9_7fa7_48ac_a05b_3cd783aee9da.slice/crio-c4598f072975afb148fdd7a4bdea1dd5572905787bb48e8a8f473b2fe429ee33 WatchSource:0}: Error finding container c4598f072975afb148fdd7a4bdea1dd5572905787bb48e8a8f473b2fe429ee33: Status 404 returned error can't find the container with id c4598f072975afb148fdd7a4bdea1dd5572905787bb48e8a8f473b2fe429ee33
Oct 30 01:08:23.686878 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:23.686849    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd"]
Oct 30 01:08:24.318834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:24.318587    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd" event={"ID":"7aa50df9-7fa7-48ac-a05b-3cd783aee9da","Type":"ContainerStarted","Data":"ed4056670b3ebe61eedbe6c3256947fcced9dcfe7abd89e769efd0046e189881"}
Oct 30 01:08:24.318834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:24.318663    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd" event={"ID":"7aa50df9-7fa7-48ac-a05b-3cd783aee9da","Type":"ContainerStarted","Data":"c4598f072975afb148fdd7a4bdea1dd5572905787bb48e8a8f473b2fe429ee33"}
Oct 30 01:08:24.358738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:24.358659    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd" podStartSLOduration=1.3586422439999999 podStartE2EDuration="1.358642244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:24.358409675 +0000 UTC m=+1998.215317430" watchObservedRunningTime="2026-10-30 01:08:24.358642244 +0000 UTC m=+1998.215550003"
Oct 30 01:08:25.339258 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:25.339218    6099 generic.go:356] "Generic (PLEG): container finished" podID="7aa50df9-7fa7-48ac-a05b-3cd783aee9da" containerID="ed4056670b3ebe61eedbe6c3256947fcced9dcfe7abd89e769efd0046e189881" exitCode=0
Oct 30 01:08:25.339776 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:25.339426    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd" event={"ID":"7aa50df9-7fa7-48ac-a05b-3cd783aee9da","Type":"ContainerDied","Data":"ed4056670b3ebe61eedbe6c3256947fcced9dcfe7abd89e769efd0046e189881"}
Oct 30 01:08:26.522950 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:26.522924    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd"
Oct 30 01:08:26.558285 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:26.558254    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-kube-api-access-zr8sh\" (UniqueName: \"kubernetes.io/projected/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-kube-api-access-zr8sh\") pod \"7aa50df9-7fa7-48ac-a05b-3cd783aee9da\" (UID: \"7aa50df9-7fa7-48ac-a05b-3cd783aee9da\") "
Oct 30 01:08:26.558588 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:26.558569    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-maas-api-xks-ca\") pod \"7aa50df9-7fa7-48ac-a05b-3cd783aee9da\" (UID: \"7aa50df9-7fa7-48ac-a05b-3cd783aee9da\") "
Oct 30 01:08:26.558875 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:26.558856    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-maas-api-openshift-ca\") pod \"7aa50df9-7fa7-48ac-a05b-3cd783aee9da\" (UID: \"7aa50df9-7fa7-48ac-a05b-3cd783aee9da\") "
Oct 30 01:08:26.560000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:26.559978    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-maas-api-openshift-ca" pod "7aa50df9-7fa7-48ac-a05b-3cd783aee9da" (UID: "7aa50df9-7fa7-48ac-a05b-3cd783aee9da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:26.583340 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:26.582236    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-kube-api-access-zr8sh" pod "7aa50df9-7fa7-48ac-a05b-3cd783aee9da" (UID: "7aa50df9-7fa7-48ac-a05b-3cd783aee9da"). InnerVolumeSpecName "kube-api-access-zr8sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:26.584043 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:26.583990    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-maas-api-xks-ca" pod "7aa50df9-7fa7-48ac-a05b-3cd783aee9da" (UID: "7aa50df9-7fa7-48ac-a05b-3cd783aee9da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:26.666886 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:26.666839    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr8sh\" (UniqueName: \"kubernetes.io/projected/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-kube-api-access-zr8sh\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:26.666886 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:26.666887    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:26.667177 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:26.666901    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa50df9-7fa7-48ac-a05b-3cd783aee9da-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:27.357226 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:27.357197    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd"
Oct 30 01:08:27.357421 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:27.357234    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd" event={"ID":"7aa50df9-7fa7-48ac-a05b-3cd783aee9da","Type":"ContainerDied","Data":"c4598f072975afb148fdd7a4bdea1dd5572905787bb48e8a8f473b2fe429ee33"}
Oct 30 01:08:27.357421 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:27.357279    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4598f072975afb148fdd7a4bdea1dd5572905787bb48e8a8f473b2fe429ee33"
Oct 30 01:08:33.351727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:33.349936    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s"
Oct 30 01:08:33.351727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:33.350922    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx"
Oct 30 01:08:34.099024 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:34.098988    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8"]
Oct 30 01:08:34.104345 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:34.104315    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-820c04-nmml8"]
Oct 30 01:08:34.265890 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:34.265856    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc93c165-35ad-4bac-9b4a-5c68e4020bfa" path="/var/lib/kubelet/pods/dc93c165-35ad-4bac-9b4a-5c68e4020bfa/volumes"
Oct 30 01:08:36.853258 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:36.853223    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8"]
Oct 30 01:08:36.856253 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:36.856226    6099 cpu_manager.go:414] "containerMap: removing container" podUID="7aa50df9-7fa7-48ac-a05b-3cd783aee9da" containerName="revoke-keys"
Oct 30 01:08:36.856253 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:36.856250    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa50df9-7fa7-48ac-a05b-3cd783aee9da" containerName="revoke-keys"
Oct 30 01:08:36.856418 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:36.856398    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa50df9-7fa7-48ac-a05b-3cd783aee9da" containerName="revoke-keys"
Oct 30 01:08:36.860531 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:36.860511    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8"
Oct 30 01:08:36.878660 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:36.878632    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8"]
Oct 30 01:08:36.884957 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:36.884922    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-361af8cf-75zz8\" (UID: \"851b0e85-b868-4d95-9ffa-ccf46d2cf31b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8"
Oct 30 01:08:36.885095 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:36.885023    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrxbv\" (UniqueName: \"kubernetes.io/projected/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-kube-api-access-rrxbv\") pod \"maas-api-revoke-sub-361af8cf-75zz8\" (UID: \"851b0e85-b868-4d95-9ffa-ccf46d2cf31b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8"
Oct 30 01:08:36.885245 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:36.885214    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-361af8cf-75zz8\" (UID: \"851b0e85-b868-4d95-9ffa-ccf46d2cf31b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8"
Oct 30 01:08:36.885412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:36.885390    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:08:36.885544 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:36.885522    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:08:36.986450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:36.986412    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-361af8cf-75zz8\" (UID: \"851b0e85-b868-4d95-9ffa-ccf46d2cf31b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8"
Oct 30 01:08:36.986630 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:36.986485    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrxbv\" (UniqueName: \"kubernetes.io/projected/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-kube-api-access-rrxbv\") pod \"maas-api-revoke-sub-361af8cf-75zz8\" (UID: \"851b0e85-b868-4d95-9ffa-ccf46d2cf31b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8"
Oct 30 01:08:36.986807 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:36.986639    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-361af8cf-75zz8\" (UID: \"851b0e85-b868-4d95-9ffa-ccf46d2cf31b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8"
Oct 30 01:08:36.989332 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:36.987315    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-361af8cf-75zz8\" (UID: \"851b0e85-b868-4d95-9ffa-ccf46d2cf31b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8"
Oct 30 01:08:36.990070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:36.990046    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-361af8cf-75zz8\" (UID: \"851b0e85-b868-4d95-9ffa-ccf46d2cf31b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8"
Oct 30 01:08:37.009785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:37.009755    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrxbv\" (UniqueName: \"kubernetes.io/projected/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-kube-api-access-rrxbv\") pod \"maas-api-revoke-sub-361af8cf-75zz8\" (UID: \"851b0e85-b868-4d95-9ffa-ccf46d2cf31b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8"
Oct 30 01:08:37.183883 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:37.183787    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8"
Oct 30 01:08:37.490738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:37.490698    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8"]
Oct 30 01:08:37.491448 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:08:37.491425    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851b0e85_b868_4d95_9ffa_ccf46d2cf31b.slice/crio-8d1468e019683ad241eeda882e28a3e10f8c1c28a7b3a963a5d59859b634b5eb WatchSource:0}: Error finding container 8d1468e019683ad241eeda882e28a3e10f8c1c28a7b3a963a5d59859b634b5eb: Status 404 returned error can't find the container with id 8d1468e019683ad241eeda882e28a3e10f8c1c28a7b3a963a5d59859b634b5eb
Oct 30 01:08:38.477064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:38.477030    6099 generic.go:356] "Generic (PLEG): container finished" podID="851b0e85-b868-4d95-9ffa-ccf46d2cf31b" containerID="5142a8d23d62dcd6141bc1ec047dc4dff125320b58f18323d0c59069436f6d1f" exitCode=0
Oct 30 01:08:38.477465 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:38.477114    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8" event={"ID":"851b0e85-b868-4d95-9ffa-ccf46d2cf31b","Type":"ContainerDied","Data":"5142a8d23d62dcd6141bc1ec047dc4dff125320b58f18323d0c59069436f6d1f"}
Oct 30 01:08:38.477465 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:38.477182    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8" event={"ID":"851b0e85-b868-4d95-9ffa-ccf46d2cf31b","Type":"ContainerStarted","Data":"8d1468e019683ad241eeda882e28a3e10f8c1c28a7b3a963a5d59859b634b5eb"}
Oct 30 01:08:39.440123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.440086    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"]
Oct 30 01:08:39.440332 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.440310    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6" podUID="423014b8-4236-4135-8410-b53876b9a755" containerName="discovery" containerID="cri-o://51c10930304963a999b30223beea8814b57165c2bd35961095528d6d30d63979" gracePeriod=30
Oct 30 01:08:39.759060 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.758990    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8"
Oct 30 01:08:39.851636 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.851605    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-maas-api-openshift-ca\") pod \"851b0e85-b868-4d95-9ffa-ccf46d2cf31b\" (UID: \"851b0e85-b868-4d95-9ffa-ccf46d2cf31b\") "
Oct 30 01:08:39.851926 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.851900    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-maas-api-xks-ca\") pod \"851b0e85-b868-4d95-9ffa-ccf46d2cf31b\" (UID: \"851b0e85-b868-4d95-9ffa-ccf46d2cf31b\") "
Oct 30 01:08:39.852105 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.852031    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-kube-api-access-rrxbv\" (UniqueName: \"kubernetes.io/projected/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-kube-api-access-rrxbv\") pod \"851b0e85-b868-4d95-9ffa-ccf46d2cf31b\" (UID: \"851b0e85-b868-4d95-9ffa-ccf46d2cf31b\") "
Oct 30 01:08:39.852105 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.852065    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-maas-api-openshift-ca" pod "851b0e85-b868-4d95-9ffa-ccf46d2cf31b" (UID: "851b0e85-b868-4d95-9ffa-ccf46d2cf31b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:39.852920 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.852898    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:39.855101 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.855070    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-maas-api-xks-ca" pod "851b0e85-b868-4d95-9ffa-ccf46d2cf31b" (UID: "851b0e85-b868-4d95-9ffa-ccf46d2cf31b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:39.857021 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.856994    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-kube-api-access-rrxbv" pod "851b0e85-b868-4d95-9ffa-ccf46d2cf31b" (UID: "851b0e85-b868-4d95-9ffa-ccf46d2cf31b"). InnerVolumeSpecName "kube-api-access-rrxbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:39.857238 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.857222    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 01:08:39.954588 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.954537    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423014b8-4236-4135-8410-b53876b9a755-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/423014b8-4236-4135-8410-b53876b9a755-local-certs\") pod \"423014b8-4236-4135-8410-b53876b9a755\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") "
Oct 30 01:08:39.955105 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.954621    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/423014b8-4236-4135-8410-b53876b9a755-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/423014b8-4236-4135-8410-b53876b9a755-istio-csr-ca-configmap\") pod \"423014b8-4236-4135-8410-b53876b9a755\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") "
Oct 30 01:08:39.955105 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.954688    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-istio-csr-dns-cert\") pod \"423014b8-4236-4135-8410-b53876b9a755\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") "
Oct 30 01:08:39.955105 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.954882    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-istio-kubeconfig\") pod \"423014b8-4236-4135-8410-b53876b9a755\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") "
Oct 30 01:08:39.955105 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.954952    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423014b8-4236-4135-8410-b53876b9a755-kube-api-access-jg6m8\" (UniqueName: \"kubernetes.io/projected/423014b8-4236-4135-8410-b53876b9a755-kube-api-access-jg6m8\") pod \"423014b8-4236-4135-8410-b53876b9a755\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") "
Oct 30 01:08:39.955105 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.955060    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423014b8-4236-4135-8410-b53876b9a755-istio-csr-ca-configmap" pod "423014b8-4236-4135-8410-b53876b9a755" (UID: "423014b8-4236-4135-8410-b53876b9a755"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:39.955569 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.955150    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423014b8-4236-4135-8410-b53876b9a755-istio-token\" (UniqueName: \"kubernetes.io/projected/423014b8-4236-4135-8410-b53876b9a755-istio-token\") pod \"423014b8-4236-4135-8410-b53876b9a755\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") "
Oct 30 01:08:39.955569 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.955200    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-cacerts\" (UniqueName: \"kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-cacerts\") pod \"423014b8-4236-4135-8410-b53876b9a755\" (UID: \"423014b8-4236-4135-8410-b53876b9a755\") "
Oct 30 01:08:39.956088 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.956059    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:39.956232 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.956092    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrxbv\" (UniqueName: \"kubernetes.io/projected/851b0e85-b868-4d95-9ffa-ccf46d2cf31b-kube-api-access-rrxbv\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:39.956232 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.956108    6099 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/423014b8-4236-4135-8410-b53876b9a755-istio-csr-ca-configmap\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:39.958667 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.958636    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-cacerts" pod "423014b8-4236-4135-8410-b53876b9a755" (UID: "423014b8-4236-4135-8410-b53876b9a755"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:39.959373 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.959346    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423014b8-4236-4135-8410-b53876b9a755-kube-api-access-jg6m8" pod "423014b8-4236-4135-8410-b53876b9a755" (UID: "423014b8-4236-4135-8410-b53876b9a755"). InnerVolumeSpecName "kube-api-access-jg6m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:39.959521 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.959499    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-istio-csr-dns-cert" pod "423014b8-4236-4135-8410-b53876b9a755" (UID: "423014b8-4236-4135-8410-b53876b9a755"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:39.959590 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.959515    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423014b8-4236-4135-8410-b53876b9a755-local-certs" pod "423014b8-4236-4135-8410-b53876b9a755" (UID: "423014b8-4236-4135-8410-b53876b9a755"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:39.959590 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.959530    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-istio-kubeconfig" pod "423014b8-4236-4135-8410-b53876b9a755" (UID: "423014b8-4236-4135-8410-b53876b9a755"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:39.961325 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:39.961303    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423014b8-4236-4135-8410-b53876b9a755-istio-token" pod "423014b8-4236-4135-8410-b53876b9a755" (UID: "423014b8-4236-4135-8410-b53876b9a755"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:40.058370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:40.058334    6099 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/423014b8-4236-4135-8410-b53876b9a755-local-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:40.058370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:40.058367    6099 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-istio-csr-dns-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:40.058586 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:40.058397    6099 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-istio-kubeconfig\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:40.058586 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:40.058406    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg6m8\" (UniqueName: \"kubernetes.io/projected/423014b8-4236-4135-8410-b53876b9a755-kube-api-access-jg6m8\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:40.058586 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:40.058416    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423014b8-4236-4135-8410-b53876b9a755-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:40.058586 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:40.058431    6099 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/423014b8-4236-4135-8410-b53876b9a755-cacerts\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:40.500162 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:40.500131    6099 generic.go:356] "Generic (PLEG): container finished" podID="423014b8-4236-4135-8410-b53876b9a755" containerID="51c10930304963a999b30223beea8814b57165c2bd35961095528d6d30d63979" exitCode=0
Oct 30 01:08:40.500314 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:40.500173    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"
Oct 30 01:08:40.500314 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:40.500218    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6" event={"ID":"423014b8-4236-4135-8410-b53876b9a755","Type":"ContainerDied","Data":"51c10930304963a999b30223beea8814b57165c2bd35961095528d6d30d63979"}
Oct 30 01:08:40.500314 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:40.500275    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6" event={"ID":"423014b8-4236-4135-8410-b53876b9a755","Type":"ContainerDied","Data":"9009aa363125a0d868bb9902c2b8cfd92efc5bc31d8a7cb9be252939fb7ecad1"}
Oct 30 01:08:40.500314 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:40.500304    6099 scope.go:122] "RemoveContainer" containerID="51c10930304963a999b30223beea8814b57165c2bd35961095528d6d30d63979"
Oct 30 01:08:40.502969 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:40.502119    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8" event={"ID":"851b0e85-b868-4d95-9ffa-ccf46d2cf31b","Type":"ContainerDied","Data":"8d1468e019683ad241eeda882e28a3e10f8c1c28a7b3a963a5d59859b634b5eb"}
Oct 30 01:08:40.502969 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:40.502142    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8"
Oct 30 01:08:40.502969 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:40.502162    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d1468e019683ad241eeda882e28a3e10f8c1c28a7b3a963a5d59859b634b5eb"
Oct 30 01:08:40.513749 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:40.513688    6099 scope.go:122] "RemoveContainer" containerID="51c10930304963a999b30223beea8814b57165c2bd35961095528d6d30d63979"
Oct 30 01:08:40.514278 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:08:40.514247    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51c10930304963a999b30223beea8814b57165c2bd35961095528d6d30d63979\": container with ID starting with 51c10930304963a999b30223beea8814b57165c2bd35961095528d6d30d63979 not found: ID does not exist" containerID="51c10930304963a999b30223beea8814b57165c2bd35961095528d6d30d63979"
Oct 30 01:08:40.514394 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:40.514289    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c10930304963a999b30223beea8814b57165c2bd35961095528d6d30d63979"} err="failed to get container status \"51c10930304963a999b30223beea8814b57165c2bd35961095528d6d30d63979\": rpc error: code = NotFound desc = could not find container \"51c10930304963a999b30223beea8814b57165c2bd35961095528d6d30d63979\": container with ID starting with 51c10930304963a999b30223beea8814b57165c2bd35961095528d6d30d63979 not found: ID does not exist"
Oct 30 01:08:40.553033 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:40.553007    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"]
Oct 30 01:08:40.564883 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:40.564858    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhlj6"]
Oct 30 01:08:41.415814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:41.415775    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"]
Oct 30 01:08:41.416319 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:41.416119    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44" podUID="7c6446b4-b8c2-42b3-b449-2ae9141a19dc" containerName="istio-proxy" containerID="cri-o://52be99d2493f3c23a6f7538e8b985abe12d94d2aceccc62f1ed6e1e6713c4656" gracePeriod=30
Oct 30 01:08:41.423858 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:41.423827    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"]
Oct 30 01:08:41.424105 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:41.424076    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq" podUID="3f728e81-7497-41de-874a-95b05a1c7395" containerName="istio-proxy" containerID="cri-o://5c31f514f04639615ca9fc5c33db573c6aa4a91d5ac9ba908608ba463e0b9be6" gracePeriod=30
Oct 30 01:08:41.937719 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:41.937674    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx"]
Oct 30 01:08:41.938498 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:41.938478    6099 cpu_manager.go:414] "containerMap: removing container" podUID="423014b8-4236-4135-8410-b53876b9a755" containerName="discovery"
Oct 30 01:08:41.938498 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:41.938498    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423014b8-4236-4135-8410-b53876b9a755" containerName="discovery"
Oct 30 01:08:41.938653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:41.938520    6099 cpu_manager.go:414] "containerMap: removing container" podUID="851b0e85-b868-4d95-9ffa-ccf46d2cf31b" containerName="revoke-keys"
Oct 30 01:08:41.938653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:41.938527    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="851b0e85-b868-4d95-9ffa-ccf46d2cf31b" containerName="revoke-keys"
Oct 30 01:08:41.939619 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:41.939587    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="423014b8-4236-4135-8410-b53876b9a755" containerName="discovery"
Oct 30 01:08:41.939771 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:41.939636    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="851b0e85-b868-4d95-9ffa-ccf46d2cf31b" containerName="revoke-keys"
Oct 30 01:08:41.948379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:41.947056    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx"
Oct 30 01:08:41.962508 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:41.962476    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx"]
Oct 30 01:08:42.032875 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:42.032835    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:08:42.033105 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:42.033085    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:08:42.101009 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:42.100961    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87dbd409-a029-4bcb-bf29-22fba38e0c24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx\" (UID: \"87dbd409-a029-4bcb-bf29-22fba38e0c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx"
Oct 30 01:08:42.101147 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:42.101122    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbzq8\" (UniqueName: \"kubernetes.io/projected/87dbd409-a029-4bcb-bf29-22fba38e0c24-kube-api-access-gbzq8\") pod \"maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx\" (UID: \"87dbd409-a029-4bcb-bf29-22fba38e0c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx"
Oct 30 01:08:42.101291 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:42.101270    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87dbd409-a029-4bcb-bf29-22fba38e0c24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx\" (UID: \"87dbd409-a029-4bcb-bf29-22fba38e0c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx"
Oct 30 01:08:42.202864 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:42.202781    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87dbd409-a029-4bcb-bf29-22fba38e0c24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx\" (UID: \"87dbd409-a029-4bcb-bf29-22fba38e0c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx"
Oct 30 01:08:42.203025 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:42.202951    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87dbd409-a029-4bcb-bf29-22fba38e0c24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx\" (UID: \"87dbd409-a029-4bcb-bf29-22fba38e0c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx"
Oct 30 01:08:42.203086 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:42.203038    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbzq8\" (UniqueName: \"kubernetes.io/projected/87dbd409-a029-4bcb-bf29-22fba38e0c24-kube-api-access-gbzq8\") pod \"maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx\" (UID: \"87dbd409-a029-4bcb-bf29-22fba38e0c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx"
Oct 30 01:08:42.203431 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:42.203408    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87dbd409-a029-4bcb-bf29-22fba38e0c24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx\" (UID: \"87dbd409-a029-4bcb-bf29-22fba38e0c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx"
Oct 30 01:08:42.205282 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:42.205259    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87dbd409-a029-4bcb-bf29-22fba38e0c24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx\" (UID: \"87dbd409-a029-4bcb-bf29-22fba38e0c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx"
Oct 30 01:08:42.227202 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:42.227179    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbzq8\" (UniqueName: \"kubernetes.io/projected/87dbd409-a029-4bcb-bf29-22fba38e0c24-kube-api-access-gbzq8\") pod \"maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx\" (UID: \"87dbd409-a029-4bcb-bf29-22fba38e0c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx"
Oct 30 01:08:42.263383 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:42.263352    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="423014b8-4236-4135-8410-b53876b9a755" path="/var/lib/kubelet/pods/423014b8-4236-4135-8410-b53876b9a755/volumes"
Oct 30 01:08:42.266337 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:42.266321    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx"
Oct 30 01:08:42.585767 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:08:42.585475    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87dbd409_a029_4bcb_bf29_22fba38e0c24.slice/crio-a178ebf7933740c255a3e0660f599347042438849d80907e5161f7bb076cc92e WatchSource:0}: Error finding container a178ebf7933740c255a3e0660f599347042438849d80907e5161f7bb076cc92e: Status 404 returned error can't find the container with id a178ebf7933740c255a3e0660f599347042438849d80907e5161f7bb076cc92e
Oct 30 01:08:42.585767 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:42.585693    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx"]
Oct 30 01:08:42.757136 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:42.753859    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:08:42.870923 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:42.870851    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq"]
Oct 30 01:08:42.877086 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:42.877061    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq"
Oct 30 01:08:42.914409 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:42.914372    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq"]
Oct 30 01:08:43.020984 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:43.020951    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160b4ab9-ba38-4572-bcec-c89aebb40117-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq\" (UID: \"160b4ab9-ba38-4572-bcec-c89aebb40117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq"
Oct 30 01:08:43.021184 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:43.021125    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6v2s\" (UniqueName: \"kubernetes.io/projected/160b4ab9-ba38-4572-bcec-c89aebb40117-kube-api-access-k6v2s\") pod \"maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq\" (UID: \"160b4ab9-ba38-4572-bcec-c89aebb40117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq"
Oct 30 01:08:43.021330 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:43.021312    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160b4ab9-ba38-4572-bcec-c89aebb40117-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq\" (UID: \"160b4ab9-ba38-4572-bcec-c89aebb40117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq"
Oct 30 01:08:43.123797 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:43.123692    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160b4ab9-ba38-4572-bcec-c89aebb40117-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq\" (UID: \"160b4ab9-ba38-4572-bcec-c89aebb40117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq"
Oct 30 01:08:43.123978 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:43.123899    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160b4ab9-ba38-4572-bcec-c89aebb40117-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq\" (UID: \"160b4ab9-ba38-4572-bcec-c89aebb40117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq"
Oct 30 01:08:43.124072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:43.124040    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6v2s\" (UniqueName: \"kubernetes.io/projected/160b4ab9-ba38-4572-bcec-c89aebb40117-kube-api-access-k6v2s\") pod \"maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq\" (UID: \"160b4ab9-ba38-4572-bcec-c89aebb40117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq"
Oct 30 01:08:43.124447 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:43.124420    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160b4ab9-ba38-4572-bcec-c89aebb40117-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq\" (UID: \"160b4ab9-ba38-4572-bcec-c89aebb40117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq"
Oct 30 01:08:43.126865 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:43.126841    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160b4ab9-ba38-4572-bcec-c89aebb40117-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq\" (UID: \"160b4ab9-ba38-4572-bcec-c89aebb40117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq"
Oct 30 01:08:43.148166 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:43.148133    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6v2s\" (UniqueName: \"kubernetes.io/projected/160b4ab9-ba38-4572-bcec-c89aebb40117-kube-api-access-k6v2s\") pod \"maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq\" (UID: \"160b4ab9-ba38-4572-bcec-c89aebb40117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq"
Oct 30 01:08:43.200012 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:43.199971    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq"
Oct 30 01:08:43.536834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:43.536687    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx" event={"ID":"87dbd409-a029-4bcb-bf29-22fba38e0c24","Type":"ContainerStarted","Data":"058160f1f3ac3f66a9d4c4a331493f0e048c88c5900dbe5931f60bfc0fbb96db"}
Oct 30 01:08:43.536834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:43.536798    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx" event={"ID":"87dbd409-a029-4bcb-bf29-22fba38e0c24","Type":"ContainerStarted","Data":"a178ebf7933740c255a3e0660f599347042438849d80907e5161f7bb076cc92e"}
Oct 30 01:08:43.577932 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:08:43.577901    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160b4ab9_ba38_4572_bcec_c89aebb40117.slice/crio-e4cada3f89d14b65d06c9cd9806ed1274e694e9b4e43ec728c551e78fd6c2ee8 WatchSource:0}: Error finding container e4cada3f89d14b65d06c9cd9806ed1274e694e9b4e43ec728c551e78fd6c2ee8: Status 404 returned error can't find the container with id e4cada3f89d14b65d06c9cd9806ed1274e694e9b4e43ec728c551e78fd6c2ee8
Oct 30 01:08:43.580407 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:43.580374    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq"]
Oct 30 01:08:44.547025 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:44.546990    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq" event={"ID":"160b4ab9-ba38-4572-bcec-c89aebb40117","Type":"ContainerStarted","Data":"a3d7ee417a824495fdd5b5f0386919c6b3d572dc1a4d8c78bc9b541ddd601bb0"}
Oct 30 01:08:44.547482 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:44.547039    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq" event={"ID":"160b4ab9-ba38-4572-bcec-c89aebb40117","Type":"ContainerStarted","Data":"e4cada3f89d14b65d06c9cd9806ed1274e694e9b4e43ec728c551e78fd6c2ee8"}
Oct 30 01:08:44.548771 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:44.548746    6099 generic.go:356] "Generic (PLEG): container finished" podID="87dbd409-a029-4bcb-bf29-22fba38e0c24" containerID="058160f1f3ac3f66a9d4c4a331493f0e048c88c5900dbe5931f60bfc0fbb96db" exitCode=0
Oct 30 01:08:44.548902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:44.548810    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx" event={"ID":"87dbd409-a029-4bcb-bf29-22fba38e0c24","Type":"ContainerDied","Data":"058160f1f3ac3f66a9d4c4a331493f0e048c88c5900dbe5931f60bfc0fbb96db"}
Oct 30 01:08:44.588520 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:44.588476    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx" podStartSLOduration=3.588461722 podStartE2EDuration="3.588461722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:43.635186438 +0000 UTC m=+2017.492094195" watchObservedRunningTime="2026-10-30 01:08:44.588461722 +0000 UTC m=+2018.445369502"
Oct 30 01:08:44.641701 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:44.641647    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq" podStartSLOduration=2.641633652 podStartE2EDuration="2.641633652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:44.597455489 +0000 UTC m=+2018.454363244" watchObservedRunningTime="2026-10-30 01:08:44.641633652 +0000 UTC m=+2018.498541406"
Oct 30 01:08:45.561393 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:45.561353    6099 generic.go:356] "Generic (PLEG): container finished" podID="160b4ab9-ba38-4572-bcec-c89aebb40117" containerID="a3d7ee417a824495fdd5b5f0386919c6b3d572dc1a4d8c78bc9b541ddd601bb0" exitCode=0
Oct 30 01:08:45.561892 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:45.561558    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq" event={"ID":"160b4ab9-ba38-4572-bcec-c89aebb40117","Type":"ContainerDied","Data":"a3d7ee417a824495fdd5b5f0386919c6b3d572dc1a4d8c78bc9b541ddd601bb0"}
Oct 30 01:08:45.574050 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:45.574015    6099 patch_prober.go:29] interesting pod/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:08:45.574193 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:45.574108    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44" podUID="7c6446b4-b8c2-42b3-b449-2ae9141a19dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:08:45.764759 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:45.764730    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx"
Oct 30 01:08:45.853329 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:45.853243    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87dbd409-a029-4bcb-bf29-22fba38e0c24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87dbd409-a029-4bcb-bf29-22fba38e0c24-maas-api-xks-ca\") pod \"87dbd409-a029-4bcb-bf29-22fba38e0c24\" (UID: \"87dbd409-a029-4bcb-bf29-22fba38e0c24\") "
Oct 30 01:08:45.853329 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:45.853324    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87dbd409-a029-4bcb-bf29-22fba38e0c24-kube-api-access-gbzq8\" (UniqueName: \"kubernetes.io/projected/87dbd409-a029-4bcb-bf29-22fba38e0c24-kube-api-access-gbzq8\") pod \"87dbd409-a029-4bcb-bf29-22fba38e0c24\" (UID: \"87dbd409-a029-4bcb-bf29-22fba38e0c24\") "
Oct 30 01:08:45.853559 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:45.853347    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87dbd409-a029-4bcb-bf29-22fba38e0c24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87dbd409-a029-4bcb-bf29-22fba38e0c24-maas-api-openshift-ca\") pod \"87dbd409-a029-4bcb-bf29-22fba38e0c24\" (UID: \"87dbd409-a029-4bcb-bf29-22fba38e0c24\") "
Oct 30 01:08:45.853824 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:45.853796    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87dbd409-a029-4bcb-bf29-22fba38e0c24-maas-api-openshift-ca" pod "87dbd409-a029-4bcb-bf29-22fba38e0c24" (UID: "87dbd409-a029-4bcb-bf29-22fba38e0c24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:45.854192 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:45.854174    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87dbd409-a029-4bcb-bf29-22fba38e0c24-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:45.855449 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:45.855426    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87dbd409-a029-4bcb-bf29-22fba38e0c24-maas-api-xks-ca" pod "87dbd409-a029-4bcb-bf29-22fba38e0c24" (UID: "87dbd409-a029-4bcb-bf29-22fba38e0c24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:45.855583 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:45.855515    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87dbd409-a029-4bcb-bf29-22fba38e0c24-kube-api-access-gbzq8" pod "87dbd409-a029-4bcb-bf29-22fba38e0c24" (UID: "87dbd409-a029-4bcb-bf29-22fba38e0c24"). InnerVolumeSpecName "kube-api-access-gbzq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:45.956629 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:45.956592    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87dbd409-a029-4bcb-bf29-22fba38e0c24-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:45.956809 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:45.956632    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbzq8\" (UniqueName: \"kubernetes.io/projected/87dbd409-a029-4bcb-bf29-22fba38e0c24-kube-api-access-gbzq8\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.572842 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.572337    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx" event={"ID":"87dbd409-a029-4bcb-bf29-22fba38e0c24","Type":"ContainerDied","Data":"a178ebf7933740c255a3e0660f599347042438849d80907e5161f7bb076cc92e"}
Oct 30 01:08:46.572842 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.572404    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a178ebf7933740c255a3e0660f599347042438849d80907e5161f7bb076cc92e"
Oct 30 01:08:46.572842 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.572478    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx"
Oct 30 01:08:46.575542 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.575425    6099 generic.go:356] "Generic (PLEG): container finished" podID="3f728e81-7497-41de-874a-95b05a1c7395" containerID="5c31f514f04639615ca9fc5c33db573c6aa4a91d5ac9ba908608ba463e0b9be6" exitCode=0
Oct 30 01:08:46.575693 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.575670    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq" event={"ID":"3f728e81-7497-41de-874a-95b05a1c7395","Type":"ContainerDied","Data":"5c31f514f04639615ca9fc5c33db573c6aa4a91d5ac9ba908608ba463e0b9be6"}
Oct 30 01:08:46.578085 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.578061    6099 generic.go:356] "Generic (PLEG): container finished" podID="7c6446b4-b8c2-42b3-b449-2ae9141a19dc" containerID="52be99d2493f3c23a6f7538e8b985abe12d94d2aceccc62f1ed6e1e6713c4656" exitCode=0
Oct 30 01:08:46.578431 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.578241    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44" event={"ID":"7c6446b4-b8c2-42b3-b449-2ae9141a19dc","Type":"ContainerDied","Data":"52be99d2493f3c23a6f7538e8b985abe12d94d2aceccc62f1ed6e1e6713c4656"}
Oct 30 01:08:46.578431 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.578312    6099 scope.go:122] "RemoveContainer" containerID="734491f5f7fe82ea06f75bf7eee81e6b6d172c3a1961436b460e56a17d464e0b"
Oct 30 01:08:46.661873 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:08:46.661833    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:08:46.766481 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.766451    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:08:46.769607 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.769578    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:08:46.788073 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.788023    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq"
Oct 30 01:08:46.871697 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.871662    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-credential-socket\") pod \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") "
Oct 30 01:08:46.871879 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.871751    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-workload-certs\") pod \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") "
Oct 30 01:08:46.871879 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.871777    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-credential-socket\") pod \"3f728e81-7497-41de-874a-95b05a1c7395\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") "
Oct 30 01:08:46.871879 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.871818    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f728e81-7497-41de-874a-95b05a1c7395-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f728e81-7497-41de-874a-95b05a1c7395-istiod-ca-cert\") pod \"3f728e81-7497-41de-874a-95b05a1c7395\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") "
Oct 30 01:08:46.871879 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.871875    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/160b4ab9-ba38-4572-bcec-c89aebb40117-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160b4ab9-ba38-4572-bcec-c89aebb40117-maas-api-openshift-ca\") pod \"160b4ab9-ba38-4572-bcec-c89aebb40117\" (UID: \"160b4ab9-ba38-4572-bcec-c89aebb40117\") "
Oct 30 01:08:46.872073 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.871921    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160b4ab9-ba38-4572-bcec-c89aebb40117-kube-api-access-k6v2s\" (UniqueName: \"kubernetes.io/projected/160b4ab9-ba38-4572-bcec-c89aebb40117-kube-api-access-k6v2s\") pod \"160b4ab9-ba38-4572-bcec-c89aebb40117\" (UID: \"160b4ab9-ba38-4572-bcec-c89aebb40117\") "
Oct 30 01:08:46.872073 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872000    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-istio-envoy\") pod \"3f728e81-7497-41de-874a-95b05a1c7395\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") "
Oct 30 01:08:46.872073 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872007    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-credential-socket" pod "7c6446b4-b8c2-42b3-b449-2ae9141a19dc" (UID: "7c6446b4-b8c2-42b3-b449-2ae9141a19dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:46.872073 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872036    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-workload-socket\") pod \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") "
Oct 30 01:08:46.872284 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872081    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-workload-socket\") pod \"3f728e81-7497-41de-874a-95b05a1c7395\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") "
Oct 30 01:08:46.872284 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872113    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-envoy\") pod \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") "
Oct 30 01:08:46.872284 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872035    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-credential-socket" pod "3f728e81-7497-41de-874a-95b05a1c7395" (UID: "3f728e81-7497-41de-874a-95b05a1c7395"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:46.872284 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872148    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-workload-certs" pod "7c6446b4-b8c2-42b3-b449-2ae9141a19dc" (UID: "7c6446b4-b8c2-42b3-b449-2ae9141a19dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:46.872284 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872168    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f728e81-7497-41de-874a-95b05a1c7395-istio-token\" (UniqueName: \"kubernetes.io/projected/3f728e81-7497-41de-874a-95b05a1c7395-istio-token\") pod \"3f728e81-7497-41de-874a-95b05a1c7395\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") "
Oct 30 01:08:46.872284 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872209    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istiod-ca-cert\") pod \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") "
Oct 30 01:08:46.872638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872300    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/160b4ab9-ba38-4572-bcec-c89aebb40117-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160b4ab9-ba38-4572-bcec-c89aebb40117-maas-api-xks-ca\") pod \"160b4ab9-ba38-4572-bcec-c89aebb40117\" (UID: \"160b4ab9-ba38-4572-bcec-c89aebb40117\") "
Oct 30 01:08:46.872638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872319    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-workload-socket" pod "3f728e81-7497-41de-874a-95b05a1c7395" (UID: "3f728e81-7497-41de-874a-95b05a1c7395"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:46.872638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872362    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-workload-certs\") pod \"3f728e81-7497-41de-874a-95b05a1c7395\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") "
Oct 30 01:08:46.872638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872414    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-workload-socket" pod "7c6446b4-b8c2-42b3-b449-2ae9141a19dc" (UID: "7c6446b4-b8c2-42b3-b449-2ae9141a19dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:46.872638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872418    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f728e81-7497-41de-874a-95b05a1c7395-istiod-ca-cert" pod "3f728e81-7497-41de-874a-95b05a1c7395" (UID: "3f728e81-7497-41de-874a-95b05a1c7395"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:46.872638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872531    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-data\") pod \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") "
Oct 30 01:08:46.872638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872580    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f728e81-7497-41de-874a-95b05a1c7395-kube-api-access-4nvlz\" (UniqueName: \"kubernetes.io/projected/3f728e81-7497-41de-874a-95b05a1c7395-kube-api-access-4nvlz\") pod \"3f728e81-7497-41de-874a-95b05a1c7395\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") "
Oct 30 01:08:46.873049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872642    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istiod-ca-cert" pod "7c6446b4-b8c2-42b3-b449-2ae9141a19dc" (UID: "7c6446b4-b8c2-42b3-b449-2ae9141a19dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:46.873049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872650    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160b4ab9-ba38-4572-bcec-c89aebb40117-maas-api-openshift-ca" pod "160b4ab9-ba38-4572-bcec-c89aebb40117" (UID: "160b4ab9-ba38-4572-bcec-c89aebb40117"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:46.873049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872662    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-istio-data\") pod \"3f728e81-7497-41de-874a-95b05a1c7395\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") "
Oct 30 01:08:46.873049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872732    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-podinfo\") pod \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") "
Oct 30 01:08:46.873049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872759    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-token\" (UniqueName: \"kubernetes.io/projected/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-token\") pod \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") "
Oct 30 01:08:46.873049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872789    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-kube-api-access-nctt7\" (UniqueName: \"kubernetes.io/projected/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-kube-api-access-nctt7\") pod \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\" (UID: \"7c6446b4-b8c2-42b3-b449-2ae9141a19dc\") "
Oct 30 01:08:46.873049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872822    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f728e81-7497-41de-874a-95b05a1c7395-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f728e81-7497-41de-874a-95b05a1c7395-istio-podinfo\") pod \"3f728e81-7497-41de-874a-95b05a1c7395\" (UID: \"3f728e81-7497-41de-874a-95b05a1c7395\") "
Oct 30 01:08:46.873049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.872969    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-data" pod "7c6446b4-b8c2-42b3-b449-2ae9141a19dc" (UID: "7c6446b4-b8c2-42b3-b449-2ae9141a19dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:46.873506 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.873167    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-istio-data" pod "3f728e81-7497-41de-874a-95b05a1c7395" (UID: "3f728e81-7497-41de-874a-95b05a1c7395"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:46.873869 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.873820    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.873869 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.873842    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-workload-certs" pod "3f728e81-7497-41de-874a-95b05a1c7395" (UID: "3f728e81-7497-41de-874a-95b05a1c7395"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:46.873869 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.873852    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.874134 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.873888    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.874134 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.873907    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.874134 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.873924    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f728e81-7497-41de-874a-95b05a1c7395-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.874134 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.873939    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160b4ab9-ba38-4572-bcec-c89aebb40117-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.874134 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.873953    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.874134 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.873967    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.874134 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.873982    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.874134 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.873996    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.876542 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.876376    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f728e81-7497-41de-874a-95b05a1c7395-istio-token" pod "3f728e81-7497-41de-874a-95b05a1c7395" (UID: "3f728e81-7497-41de-874a-95b05a1c7395"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:46.876542 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.876508    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-kube-api-access-nctt7" pod "7c6446b4-b8c2-42b3-b449-2ae9141a19dc" (UID: "7c6446b4-b8c2-42b3-b449-2ae9141a19dc"). InnerVolumeSpecName "kube-api-access-nctt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:46.877100 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.877076    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-envoy" pod "7c6446b4-b8c2-42b3-b449-2ae9141a19dc" (UID: "7c6446b4-b8c2-42b3-b449-2ae9141a19dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:46.877100 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.877090    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-istio-envoy" pod "3f728e81-7497-41de-874a-95b05a1c7395" (UID: "3f728e81-7497-41de-874a-95b05a1c7395"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:46.877309 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.877191    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/160b4ab9-ba38-4572-bcec-c89aebb40117-maas-api-xks-ca" pod "160b4ab9-ba38-4572-bcec-c89aebb40117" (UID: "160b4ab9-ba38-4572-bcec-c89aebb40117"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:46.877615 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.877572    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f728e81-7497-41de-874a-95b05a1c7395-kube-api-access-4nvlz" pod "3f728e81-7497-41de-874a-95b05a1c7395" (UID: "3f728e81-7497-41de-874a-95b05a1c7395"). InnerVolumeSpecName "kube-api-access-4nvlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:46.877615 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.877584    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-token" pod "7c6446b4-b8c2-42b3-b449-2ae9141a19dc" (UID: "7c6446b4-b8c2-42b3-b449-2ae9141a19dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:46.877753 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.877690    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160b4ab9-ba38-4572-bcec-c89aebb40117-kube-api-access-k6v2s" pod "160b4ab9-ba38-4572-bcec-c89aebb40117" (UID: "160b4ab9-ba38-4572-bcec-c89aebb40117"). InnerVolumeSpecName "kube-api-access-k6v2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:46.877889 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.877871    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-podinfo" pod "7c6446b4-b8c2-42b3-b449-2ae9141a19dc" (UID: "7c6446b4-b8c2-42b3-b449-2ae9141a19dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:08:46.879096 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.879076    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f728e81-7497-41de-874a-95b05a1c7395-istio-podinfo" pod "3f728e81-7497-41de-874a-95b05a1c7395" (UID: "3f728e81-7497-41de-874a-95b05a1c7395"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:08:46.975293 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.975258    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6v2s\" (UniqueName: \"kubernetes.io/projected/160b4ab9-ba38-4572-bcec-c89aebb40117-kube-api-access-k6v2s\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.975293 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.975290    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.975293 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.975300    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.975558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.975309    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f728e81-7497-41de-874a-95b05a1c7395-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.975558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.975318    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160b4ab9-ba38-4572-bcec-c89aebb40117-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.975558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.975326    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f728e81-7497-41de-874a-95b05a1c7395-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.975558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.975336    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nvlz\" (UniqueName: \"kubernetes.io/projected/3f728e81-7497-41de-874a-95b05a1c7395-kube-api-access-4nvlz\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.975558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.975345    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.975558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.975354    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.975558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.975365    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nctt7\" (UniqueName: \"kubernetes.io/projected/7c6446b4-b8c2-42b3-b449-2ae9141a19dc-kube-api-access-nctt7\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:46.975558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:46.975373    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f728e81-7497-41de-874a-95b05a1c7395-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:47.588699 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:47.588666    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"
Oct 30 01:08:47.589220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:47.588687    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq" event={"ID":"3f728e81-7497-41de-874a-95b05a1c7395","Type":"ContainerDied","Data":"304c9a6888e0edcc950d13c42864c5840ef5dca2ca9b43a92e87255136609610"}
Oct 30 01:08:47.589220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:47.588789    6099 scope.go:122] "RemoveContainer" containerID="5c31f514f04639615ca9fc5c33db573c6aa4a91d5ac9ba908608ba463e0b9be6"
Oct 30 01:08:47.590376 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:47.590347    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq"
Oct 30 01:08:47.590498 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:47.590379    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq" event={"ID":"160b4ab9-ba38-4572-bcec-c89aebb40117","Type":"ContainerDied","Data":"e4cada3f89d14b65d06c9cd9806ed1274e694e9b4e43ec728c551e78fd6c2ee8"}
Oct 30 01:08:47.590498 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:47.590418    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4cada3f89d14b65d06c9cd9806ed1274e694e9b4e43ec728c551e78fd6c2ee8"
Oct 30 01:08:47.596275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:47.596248    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"
Oct 30 01:08:47.596378 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:47.596314    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44" event={"ID":"7c6446b4-b8c2-42b3-b449-2ae9141a19dc","Type":"ContainerDied","Data":"7299ee14f309a39a0e2382bbee4b2ea2436d590088992aa34bea72b4ed263768"}
Oct 30 01:08:47.605439 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:47.605405    6099 scope.go:122] "RemoveContainer" containerID="52be99d2493f3c23a6f7538e8b985abe12d94d2aceccc62f1ed6e1e6713c4656"
Oct 30 01:08:47.658887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:47.658860    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"]
Oct 30 01:08:47.662576 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:47.662540    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-t8k44"]
Oct 30 01:08:47.694592 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:47.694546    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"]
Oct 30 01:08:47.698394 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:47.698324    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-820c04-openshift-default-bc774f4f-k2sxq"]
Oct 30 01:08:48.272822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.272778    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f728e81-7497-41de-874a-95b05a1c7395" path="/var/lib/kubelet/pods/3f728e81-7497-41de-874a-95b05a1c7395/volumes"
Oct 30 01:08:48.273766 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.273694    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c6446b4-b8c2-42b3-b449-2ae9141a19dc" path="/var/lib/kubelet/pods/7c6446b4-b8c2-42b3-b449-2ae9141a19dc/volumes"
Oct 30 01:08:48.554437 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.554404    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp"]
Oct 30 01:08:48.555156 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.555141    6099 cpu_manager.go:414] "containerMap: removing container" podUID="3f728e81-7497-41de-874a-95b05a1c7395" containerName="istio-proxy"
Oct 30 01:08:48.555204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.555158    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f728e81-7497-41de-874a-95b05a1c7395" containerName="istio-proxy"
Oct 30 01:08:48.555204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.555176    6099 cpu_manager.go:414] "containerMap: removing container" podUID="7c6446b4-b8c2-42b3-b449-2ae9141a19dc" containerName="istio-proxy"
Oct 30 01:08:48.555204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.555181    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6446b4-b8c2-42b3-b449-2ae9141a19dc" containerName="istio-proxy"
Oct 30 01:08:48.555204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.555197    6099 cpu_manager.go:414] "containerMap: removing container" podUID="160b4ab9-ba38-4572-bcec-c89aebb40117" containerName="revoke-keys"
Oct 30 01:08:48.555204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.555202    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160b4ab9-ba38-4572-bcec-c89aebb40117" containerName="revoke-keys"
Oct 30 01:08:48.555365 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.555220    6099 cpu_manager.go:414] "containerMap: removing container" podUID="87dbd409-a029-4bcb-bf29-22fba38e0c24" containerName="revoke-keys"
Oct 30 01:08:48.555365 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.555226    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87dbd409-a029-4bcb-bf29-22fba38e0c24" containerName="revoke-keys"
Oct 30 01:08:48.555365 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.555364    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6446b4-b8c2-42b3-b449-2ae9141a19dc" containerName="istio-proxy"
Oct 30 01:08:48.555473 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.555376    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f728e81-7497-41de-874a-95b05a1c7395" containerName="istio-proxy"
Oct 30 01:08:48.555473 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.555384    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6446b4-b8c2-42b3-b449-2ae9141a19dc" containerName="istio-proxy"
Oct 30 01:08:48.555473 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.555393    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="160b4ab9-ba38-4572-bcec-c89aebb40117" containerName="revoke-keys"
Oct 30 01:08:48.555473 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.555402    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="87dbd409-a029-4bcb-bf29-22fba38e0c24" containerName="revoke-keys"
Oct 30 01:08:48.560656 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.560641    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp"
Oct 30 01:08:48.578794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.575315    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:08:48.578794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.575338    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:08:48.578794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.577844    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp"]
Oct 30 01:08:48.600796 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.600755    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5vr8\" (UniqueName: \"kubernetes.io/projected/89fa8089-4f05-4736-93c5-42f3362cf38c-kube-api-access-t5vr8\") pod \"maas-api-revoke-sub-9656d13e-bm7bp\" (UID: \"89fa8089-4f05-4736-93c5-42f3362cf38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp"
Oct 30 01:08:48.601244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.600878    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89fa8089-4f05-4736-93c5-42f3362cf38c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9656d13e-bm7bp\" (UID: \"89fa8089-4f05-4736-93c5-42f3362cf38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp"
Oct 30 01:08:48.601244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.601039    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89fa8089-4f05-4736-93c5-42f3362cf38c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9656d13e-bm7bp\" (UID: \"89fa8089-4f05-4736-93c5-42f3362cf38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp"
Oct 30 01:08:48.703101 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.703066    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5vr8\" (UniqueName: \"kubernetes.io/projected/89fa8089-4f05-4736-93c5-42f3362cf38c-kube-api-access-t5vr8\") pod \"maas-api-revoke-sub-9656d13e-bm7bp\" (UID: \"89fa8089-4f05-4736-93c5-42f3362cf38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp"
Oct 30 01:08:48.703272 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.703129    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89fa8089-4f05-4736-93c5-42f3362cf38c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9656d13e-bm7bp\" (UID: \"89fa8089-4f05-4736-93c5-42f3362cf38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp"
Oct 30 01:08:48.703341 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.703315    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89fa8089-4f05-4736-93c5-42f3362cf38c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9656d13e-bm7bp\" (UID: \"89fa8089-4f05-4736-93c5-42f3362cf38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp"
Oct 30 01:08:48.703902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.703881    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89fa8089-4f05-4736-93c5-42f3362cf38c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9656d13e-bm7bp\" (UID: \"89fa8089-4f05-4736-93c5-42f3362cf38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp"
Oct 30 01:08:48.705612 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.705593    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89fa8089-4f05-4736-93c5-42f3362cf38c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9656d13e-bm7bp\" (UID: \"89fa8089-4f05-4736-93c5-42f3362cf38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp"
Oct 30 01:08:48.726054 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.726023    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5vr8\" (UniqueName: \"kubernetes.io/projected/89fa8089-4f05-4736-93c5-42f3362cf38c-kube-api-access-t5vr8\") pod \"maas-api-revoke-sub-9656d13e-bm7bp\" (UID: \"89fa8089-4f05-4736-93c5-42f3362cf38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp"
Oct 30 01:08:48.874971 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:48.874895    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp"
Oct 30 01:08:49.182326 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:08:49.182292    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fa8089_4f05_4736_93c5_42f3362cf38c.slice/crio-54fd1becd28d297a10f94a5bbbc6a5d32612089acac3182929250b3bfe28b24d WatchSource:0}: Error finding container 54fd1becd28d297a10f94a5bbbc6a5d32612089acac3182929250b3bfe28b24d: Status 404 returned error can't find the container with id 54fd1becd28d297a10f94a5bbbc6a5d32612089acac3182929250b3bfe28b24d
Oct 30 01:08:49.182559 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:49.182537    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp"]
Oct 30 01:08:49.620168 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:49.620132    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp" event={"ID":"89fa8089-4f05-4736-93c5-42f3362cf38c","Type":"ContainerStarted","Data":"50711919164e923856c921288125aeb8e9bb849946e47009b2207ad359b6f4ad"}
Oct 30 01:08:49.620615 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:49.620187    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp" event={"ID":"89fa8089-4f05-4736-93c5-42f3362cf38c","Type":"ContainerStarted","Data":"54fd1becd28d297a10f94a5bbbc6a5d32612089acac3182929250b3bfe28b24d"}
Oct 30 01:08:49.661909 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:49.661857    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp" podStartSLOduration=1.6618442770000001 podStartE2EDuration="1.661844277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:49.659085752 +0000 UTC m=+2023.515993510" watchObservedRunningTime="2026-10-30 01:08:49.661844277 +0000 UTC m=+2023.518752028"
Oct 30 01:08:50.631723 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:50.631663    6099 generic.go:356] "Generic (PLEG): container finished" podID="89fa8089-4f05-4736-93c5-42f3362cf38c" containerID="50711919164e923856c921288125aeb8e9bb849946e47009b2207ad359b6f4ad" exitCode=0
Oct 30 01:08:50.632183 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:50.631746    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp" event={"ID":"89fa8089-4f05-4736-93c5-42f3362cf38c","Type":"ContainerDied","Data":"50711919164e923856c921288125aeb8e9bb849946e47009b2207ad359b6f4ad"}
Oct 30 01:08:51.749452 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:51.749416    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"]
Oct 30 01:08:51.751050 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:51.749684    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw" podUID="1c02eae9-c5d9-459d-8969-b6d3686999d3" containerName="istio-proxy" containerID="cri-o://371a6aa0a2fa7d8d332cb51ca58472d9965cebf3400f092c0164aa6acc98aea5" gracePeriod=30
Oct 30 01:08:51.842788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:51.842758    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"]
Oct 30 01:08:51.843233 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:51.843201    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp" podUID="74025dac-eaf2-43b0-82d0-5470bd908c97" containerName="istio-proxy" containerID="cri-o://63354bb26bba83abd609bc2456a2f383061ae431923ea8dd348e91e375a0ed7c" gracePeriod=30
Oct 30 01:08:51.863326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:51.863293    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp"
Oct 30 01:08:51.968783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:51.961791    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89fa8089-4f05-4736-93c5-42f3362cf38c-kube-api-access-t5vr8\" (UniqueName: \"kubernetes.io/projected/89fa8089-4f05-4736-93c5-42f3362cf38c-kube-api-access-t5vr8\") pod \"89fa8089-4f05-4736-93c5-42f3362cf38c\" (UID: \"89fa8089-4f05-4736-93c5-42f3362cf38c\") "
Oct 30 01:08:51.968783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:51.961897    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89fa8089-4f05-4736-93c5-42f3362cf38c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89fa8089-4f05-4736-93c5-42f3362cf38c-maas-api-xks-ca\") pod \"89fa8089-4f05-4736-93c5-42f3362cf38c\" (UID: \"89fa8089-4f05-4736-93c5-42f3362cf38c\") "
Oct 30 01:08:51.968783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:51.961948    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89fa8089-4f05-4736-93c5-42f3362cf38c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89fa8089-4f05-4736-93c5-42f3362cf38c-maas-api-openshift-ca\") pod \"89fa8089-4f05-4736-93c5-42f3362cf38c\" (UID: \"89fa8089-4f05-4736-93c5-42f3362cf38c\") "
Oct 30 01:08:51.968783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:51.967021    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89fa8089-4f05-4736-93c5-42f3362cf38c-maas-api-openshift-ca" pod "89fa8089-4f05-4736-93c5-42f3362cf38c" (UID: "89fa8089-4f05-4736-93c5-42f3362cf38c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:51.999802 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:51.999766    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x"]
Oct 30 01:08:52.001159 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.001131    6099 cpu_manager.go:414] "containerMap: removing container" podUID="89fa8089-4f05-4736-93c5-42f3362cf38c" containerName="revoke-keys"
Oct 30 01:08:52.001326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.001311    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89fa8089-4f05-4736-93c5-42f3362cf38c" containerName="revoke-keys"
Oct 30 01:08:52.001489 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.001477    6099 cpu_manager.go:414] "containerMap: removing container" podUID="7c6446b4-b8c2-42b3-b449-2ae9141a19dc" containerName="istio-proxy"
Oct 30 01:08:52.001574 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.001562    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6446b4-b8c2-42b3-b449-2ae9141a19dc" containerName="istio-proxy"
Oct 30 01:08:52.001898 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.001880    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="89fa8089-4f05-4736-93c5-42f3362cf38c" containerName="revoke-keys"
Oct 30 01:08:52.040575 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.037989    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x"
Oct 30 01:08:52.056964 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.052751    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89fa8089-4f05-4736-93c5-42f3362cf38c-maas-api-xks-ca" pod "89fa8089-4f05-4736-93c5-42f3362cf38c" (UID: "89fa8089-4f05-4736-93c5-42f3362cf38c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:52.097753 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.097695    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad36020-3ebe-43b5-b83e-e996bc66e201-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-780077-cgn7x\" (UID: \"bad36020-3ebe-43b5-b83e-e996bc66e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x"
Oct 30 01:08:52.098105 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.098085    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8kqc\" (UniqueName: \"kubernetes.io/projected/bad36020-3ebe-43b5-b83e-e996bc66e201-kube-api-access-j8kqc\") pod \"maas-api-revoke-keys-e2e-inf-a-780077-cgn7x\" (UID: \"bad36020-3ebe-43b5-b83e-e996bc66e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x"
Oct 30 01:08:52.098400 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.098383    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad36020-3ebe-43b5-b83e-e996bc66e201-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-780077-cgn7x\" (UID: \"bad36020-3ebe-43b5-b83e-e996bc66e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x"
Oct 30 01:08:52.100023 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.099999    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89fa8089-4f05-4736-93c5-42f3362cf38c-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:52.100173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.100161    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89fa8089-4f05-4736-93c5-42f3362cf38c-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:52.119174 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.112451    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x"]
Oct 30 01:08:52.119174 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.113329    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89fa8089-4f05-4736-93c5-42f3362cf38c-kube-api-access-t5vr8" pod "89fa8089-4f05-4736-93c5-42f3362cf38c" (UID: "89fa8089-4f05-4736-93c5-42f3362cf38c"). InnerVolumeSpecName "kube-api-access-t5vr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:52.201141 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.201106    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad36020-3ebe-43b5-b83e-e996bc66e201-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-780077-cgn7x\" (UID: \"bad36020-3ebe-43b5-b83e-e996bc66e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x"
Oct 30 01:08:52.201332 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.201230    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8kqc\" (UniqueName: \"kubernetes.io/projected/bad36020-3ebe-43b5-b83e-e996bc66e201-kube-api-access-j8kqc\") pod \"maas-api-revoke-keys-e2e-inf-a-780077-cgn7x\" (UID: \"bad36020-3ebe-43b5-b83e-e996bc66e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x"
Oct 30 01:08:52.201441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.201424    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad36020-3ebe-43b5-b83e-e996bc66e201-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-780077-cgn7x\" (UID: \"bad36020-3ebe-43b5-b83e-e996bc66e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x"
Oct 30 01:08:52.201614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.201600    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5vr8\" (UniqueName: \"kubernetes.io/projected/89fa8089-4f05-4736-93c5-42f3362cf38c-kube-api-access-t5vr8\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:52.201765 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.201746    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad36020-3ebe-43b5-b83e-e996bc66e201-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-780077-cgn7x\" (UID: \"bad36020-3ebe-43b5-b83e-e996bc66e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x"
Oct 30 01:08:52.203684 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.203664    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad36020-3ebe-43b5-b83e-e996bc66e201-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-780077-cgn7x\" (UID: \"bad36020-3ebe-43b5-b83e-e996bc66e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x"
Oct 30 01:08:52.223932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.223881    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8kqc\" (UniqueName: \"kubernetes.io/projected/bad36020-3ebe-43b5-b83e-e996bc66e201-kube-api-access-j8kqc\") pod \"maas-api-revoke-keys-e2e-inf-a-780077-cgn7x\" (UID: \"bad36020-3ebe-43b5-b83e-e996bc66e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x"
Oct 30 01:08:52.450012 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.449977    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x"
Oct 30 01:08:52.648236 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.648208    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp"
Oct 30 01:08:52.648399 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.648241    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp" event={"ID":"89fa8089-4f05-4736-93c5-42f3362cf38c","Type":"ContainerDied","Data":"54fd1becd28d297a10f94a5bbbc6a5d32612089acac3182929250b3bfe28b24d"}
Oct 30 01:08:52.648399 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.648292    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54fd1becd28d297a10f94a5bbbc6a5d32612089acac3182929250b3bfe28b24d"
Oct 30 01:08:52.767553 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:52.767516    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x"]
Oct 30 01:08:52.776828 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:08:52.776798    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad36020_3ebe_43b5_b83e_e996bc66e201.slice/crio-bcdfaea8215857d926f5e5c94a64a05005a321b872e29c64b3449bb07ecd0269 WatchSource:0}: Error finding container bcdfaea8215857d926f5e5c94a64a05005a321b872e29c64b3449bb07ecd0269: Status 404 returned error can't find the container with id bcdfaea8215857d926f5e5c94a64a05005a321b872e29c64b3449bb07ecd0269
Oct 30 01:08:53.695102 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:53.695063    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x" event={"ID":"bad36020-3ebe-43b5-b83e-e996bc66e201","Type":"ContainerStarted","Data":"06fb38e8f1617638b68530c2e8eaf1f9cf0437637bb53c09b621bc3e9394a09c"}
Oct 30 01:08:53.695504 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:53.695483    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x" event={"ID":"bad36020-3ebe-43b5-b83e-e996bc66e201","Type":"ContainerStarted","Data":"bcdfaea8215857d926f5e5c94a64a05005a321b872e29c64b3449bb07ecd0269"}
Oct 30 01:08:53.735369 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:53.735309    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x" podStartSLOduration=2.735292836 podStartE2EDuration="2.735292836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:53.734387904 +0000 UTC m=+2027.591295658" watchObservedRunningTime="2026-10-30 01:08:53.735292836 +0000 UTC m=+2027.592200592"
Oct 30 01:08:54.706019 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:54.705979    6099 generic.go:356] "Generic (PLEG): container finished" podID="bad36020-3ebe-43b5-b83e-e996bc66e201" containerID="06fb38e8f1617638b68530c2e8eaf1f9cf0437637bb53c09b621bc3e9394a09c" exitCode=0
Oct 30 01:08:54.706412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:54.706026    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x" event={"ID":"bad36020-3ebe-43b5-b83e-e996bc66e201","Type":"ContainerDied","Data":"06fb38e8f1617638b68530c2e8eaf1f9cf0437637bb53c09b621bc3e9394a09c"}
Oct 30 01:08:55.910264 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:55.910234    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x"
Oct 30 01:08:56.000911 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:56.000815    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad36020-3ebe-43b5-b83e-e996bc66e201-kube-api-access-j8kqc\" (UniqueName: \"kubernetes.io/projected/bad36020-3ebe-43b5-b83e-e996bc66e201-kube-api-access-j8kqc\") pod \"bad36020-3ebe-43b5-b83e-e996bc66e201\" (UID: \"bad36020-3ebe-43b5-b83e-e996bc66e201\") "
Oct 30 01:08:56.001098 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:56.000976    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bad36020-3ebe-43b5-b83e-e996bc66e201-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad36020-3ebe-43b5-b83e-e996bc66e201-maas-api-openshift-ca\") pod \"bad36020-3ebe-43b5-b83e-e996bc66e201\" (UID: \"bad36020-3ebe-43b5-b83e-e996bc66e201\") "
Oct 30 01:08:56.001229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:56.001209    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bad36020-3ebe-43b5-b83e-e996bc66e201-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad36020-3ebe-43b5-b83e-e996bc66e201-maas-api-xks-ca\") pod \"bad36020-3ebe-43b5-b83e-e996bc66e201\" (UID: \"bad36020-3ebe-43b5-b83e-e996bc66e201\") "
Oct 30 01:08:56.001631 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:56.001599    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad36020-3ebe-43b5-b83e-e996bc66e201-maas-api-openshift-ca" pod "bad36020-3ebe-43b5-b83e-e996bc66e201" (UID: "bad36020-3ebe-43b5-b83e-e996bc66e201"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:56.001991 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:56.001969    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad36020-3ebe-43b5-b83e-e996bc66e201-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:56.004989 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:56.004961    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad36020-3ebe-43b5-b83e-e996bc66e201-kube-api-access-j8kqc" pod "bad36020-3ebe-43b5-b83e-e996bc66e201" (UID: "bad36020-3ebe-43b5-b83e-e996bc66e201"). InnerVolumeSpecName "kube-api-access-j8kqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:56.005545 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:56.005522    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bad36020-3ebe-43b5-b83e-e996bc66e201-maas-api-xks-ca" pod "bad36020-3ebe-43b5-b83e-e996bc66e201" (UID: "bad36020-3ebe-43b5-b83e-e996bc66e201"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:56.103262 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:56.103232    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad36020-3ebe-43b5-b83e-e996bc66e201-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:56.103262 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:56.103259    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8kqc\" (UniqueName: \"kubernetes.io/projected/bad36020-3ebe-43b5-b83e-e996bc66e201-kube-api-access-j8kqc\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:56.723839 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:56.723809    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x"
Oct 30 01:08:56.724021 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:56.723850    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x" event={"ID":"bad36020-3ebe-43b5-b83e-e996bc66e201","Type":"ContainerDied","Data":"bcdfaea8215857d926f5e5c94a64a05005a321b872e29c64b3449bb07ecd0269"}
Oct 30 01:08:56.724021 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:56.723904    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcdfaea8215857d926f5e5c94a64a05005a321b872e29c64b3449bb07ecd0269"
Oct 30 01:08:57.072242 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.072209    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 01:08:57.184660 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.184613    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 01:08:57.224599 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.224570    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-workload-socket\") pod \"1c02eae9-c5d9-459d-8969-b6d3686999d3\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") "
Oct 30 01:08:57.224813 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.224608    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-token\" (UniqueName: \"kubernetes.io/projected/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-token\") pod \"1c02eae9-c5d9-459d-8969-b6d3686999d3\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") "
Oct 30 01:08:57.224813 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.224771    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-data\") pod \"1c02eae9-c5d9-459d-8969-b6d3686999d3\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") "
Oct 30 01:08:57.224952 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.224809    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c02eae9-c5d9-459d-8969-b6d3686999d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c02eae9-c5d9-459d-8969-b6d3686999d3-istiod-ca-cert\") pod \"1c02eae9-c5d9-459d-8969-b6d3686999d3\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") "
Oct 30 01:08:57.224952 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.224875    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-workload-socket" pod "1c02eae9-c5d9-459d-8969-b6d3686999d3" (UID: "1c02eae9-c5d9-459d-8969-b6d3686999d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:57.224952 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.224939    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-credential-socket\") pod \"1c02eae9-c5d9-459d-8969-b6d3686999d3\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") "
Oct 30 01:08:57.225088 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.225058    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c02eae9-c5d9-459d-8969-b6d3686999d3-kube-api-access-scwhw\" (UniqueName: \"kubernetes.io/projected/1c02eae9-c5d9-459d-8969-b6d3686999d3-kube-api-access-scwhw\") pod \"1c02eae9-c5d9-459d-8969-b6d3686999d3\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") "
Oct 30 01:08:57.225088 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.225074    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-data" pod "1c02eae9-c5d9-459d-8969-b6d3686999d3" (UID: "1c02eae9-c5d9-459d-8969-b6d3686999d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:57.225211 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.225194    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-envoy\") pod \"1c02eae9-c5d9-459d-8969-b6d3686999d3\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") "
Oct 30 01:08:57.225251 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.225192    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-credential-socket" pod "1c02eae9-c5d9-459d-8969-b6d3686999d3" (UID: "1c02eae9-c5d9-459d-8969-b6d3686999d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:57.225251 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.225218    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c02eae9-c5d9-459d-8969-b6d3686999d3-istiod-ca-cert" pod "1c02eae9-c5d9-459d-8969-b6d3686999d3" (UID: "1c02eae9-c5d9-459d-8969-b6d3686999d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:57.225332 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.225230    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-podinfo\") pod \"1c02eae9-c5d9-459d-8969-b6d3686999d3\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") "
Oct 30 01:08:57.225332 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.225322    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-workload-certs\") pod \"1c02eae9-c5d9-459d-8969-b6d3686999d3\" (UID: \"1c02eae9-c5d9-459d-8969-b6d3686999d3\") "
Oct 30 01:08:57.225654 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.225625    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-workload-certs" pod "1c02eae9-c5d9-459d-8969-b6d3686999d3" (UID: "1c02eae9-c5d9-459d-8969-b6d3686999d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:57.226065 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.226045    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:57.226160 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.226069    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c02eae9-c5d9-459d-8969-b6d3686999d3-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:57.226160 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.226082    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:57.226160 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.226094    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:57.226160 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.226108    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:57.227031 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.227010    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-token" pod "1c02eae9-c5d9-459d-8969-b6d3686999d3" (UID: "1c02eae9-c5d9-459d-8969-b6d3686999d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:57.227142 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.227109    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c02eae9-c5d9-459d-8969-b6d3686999d3-kube-api-access-scwhw" pod "1c02eae9-c5d9-459d-8969-b6d3686999d3" (UID: "1c02eae9-c5d9-459d-8969-b6d3686999d3"). InnerVolumeSpecName "kube-api-access-scwhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:57.227574 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.227554    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-podinfo" pod "1c02eae9-c5d9-459d-8969-b6d3686999d3" (UID: "1c02eae9-c5d9-459d-8969-b6d3686999d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:08:57.227692 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.227668    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-envoy" pod "1c02eae9-c5d9-459d-8969-b6d3686999d3" (UID: "1c02eae9-c5d9-459d-8969-b6d3686999d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:57.326908 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.326876    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-token\" (UniqueName: \"kubernetes.io/projected/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-token\") pod \"74025dac-eaf2-43b0-82d0-5470bd908c97\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") "
Oct 30 01:08:57.327079 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.327018    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-envoy\") pod \"74025dac-eaf2-43b0-82d0-5470bd908c97\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") "
Oct 30 01:08:57.327079 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.327065    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74025dac-eaf2-43b0-82d0-5470bd908c97-kube-api-access-vbsls\" (UniqueName: \"kubernetes.io/projected/74025dac-eaf2-43b0-82d0-5470bd908c97-kube-api-access-vbsls\") pod \"74025dac-eaf2-43b0-82d0-5470bd908c97\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") "
Oct 30 01:08:57.327191 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.327119    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-workload-certs\") pod \"74025dac-eaf2-43b0-82d0-5470bd908c97\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") "
Oct 30 01:08:57.327191 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.327150    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74025dac-eaf2-43b0-82d0-5470bd908c97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74025dac-eaf2-43b0-82d0-5470bd908c97-istiod-ca-cert\") pod \"74025dac-eaf2-43b0-82d0-5470bd908c97\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") "
Oct 30 01:08:57.327350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.327331    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-data\") pod \"74025dac-eaf2-43b0-82d0-5470bd908c97\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") "
Oct 30 01:08:57.327439 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.327403    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-credential-socket\") pod \"74025dac-eaf2-43b0-82d0-5470bd908c97\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") "
Oct 30 01:08:57.327499 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.327461    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-workload-certs" pod "74025dac-eaf2-43b0-82d0-5470bd908c97" (UID: "74025dac-eaf2-43b0-82d0-5470bd908c97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:57.327499 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.327491    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-podinfo\") pod \"74025dac-eaf2-43b0-82d0-5470bd908c97\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") "
Oct 30 01:08:57.327620 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.327532    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-workload-socket\") pod \"74025dac-eaf2-43b0-82d0-5470bd908c97\" (UID: \"74025dac-eaf2-43b0-82d0-5470bd908c97\") "
Oct 30 01:08:57.327620 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.327585    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-data" pod "74025dac-eaf2-43b0-82d0-5470bd908c97" (UID: "74025dac-eaf2-43b0-82d0-5470bd908c97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:57.327769 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.327631    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74025dac-eaf2-43b0-82d0-5470bd908c97-istiod-ca-cert" pod "74025dac-eaf2-43b0-82d0-5470bd908c97" (UID: "74025dac-eaf2-43b0-82d0-5470bd908c97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:57.327769 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.327677    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-credential-socket" pod "74025dac-eaf2-43b0-82d0-5470bd908c97" (UID: "74025dac-eaf2-43b0-82d0-5470bd908c97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:57.327925 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.327902    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-workload-socket" pod "74025dac-eaf2-43b0-82d0-5470bd908c97" (UID: "74025dac-eaf2-43b0-82d0-5470bd908c97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:57.328458 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.328433    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:57.328617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.328460    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:57.328617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.328475    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:57.328617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.328488    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:57.328617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.328503    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:57.328617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.328517    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scwhw\" (UniqueName: \"kubernetes.io/projected/1c02eae9-c5d9-459d-8969-b6d3686999d3-kube-api-access-scwhw\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:57.328617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.328531    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:57.328617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.328544    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74025dac-eaf2-43b0-82d0-5470bd908c97-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:57.328617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.328557    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c02eae9-c5d9-459d-8969-b6d3686999d3-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:57.329368 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.329344    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-envoy" pod "74025dac-eaf2-43b0-82d0-5470bd908c97" (UID: "74025dac-eaf2-43b0-82d0-5470bd908c97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:57.329479 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.329461    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-token" pod "74025dac-eaf2-43b0-82d0-5470bd908c97" (UID: "74025dac-eaf2-43b0-82d0-5470bd908c97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:57.329528 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.329510    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74025dac-eaf2-43b0-82d0-5470bd908c97-kube-api-access-vbsls" pod "74025dac-eaf2-43b0-82d0-5470bd908c97" (UID: "74025dac-eaf2-43b0-82d0-5470bd908c97"). InnerVolumeSpecName "kube-api-access-vbsls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:57.329625 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.329610    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-podinfo" pod "74025dac-eaf2-43b0-82d0-5470bd908c97" (UID: "74025dac-eaf2-43b0-82d0-5470bd908c97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:08:57.430700 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.430673    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:57.430700 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.430697    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:57.430931 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.430743    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbsls\" (UniqueName: \"kubernetes.io/projected/74025dac-eaf2-43b0-82d0-5470bd908c97-kube-api-access-vbsls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:57.430931 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.430761    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74025dac-eaf2-43b0-82d0-5470bd908c97-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:08:57.734545 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.734466    6099 generic.go:356] "Generic (PLEG): container finished" podID="1c02eae9-c5d9-459d-8969-b6d3686999d3" containerID="371a6aa0a2fa7d8d332cb51ca58472d9965cebf3400f092c0164aa6acc98aea5" exitCode=0
Oct 30 01:08:57.734545 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.734516    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"
Oct 30 01:08:57.734813 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.734539    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw" event={"ID":"1c02eae9-c5d9-459d-8969-b6d3686999d3","Type":"ContainerDied","Data":"371a6aa0a2fa7d8d332cb51ca58472d9965cebf3400f092c0164aa6acc98aea5"}
Oct 30 01:08:57.734813 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.734593    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw" event={"ID":"1c02eae9-c5d9-459d-8969-b6d3686999d3","Type":"ContainerDied","Data":"8975a69701e9b09228699c713542e200d37734fd991dc40637525c0c30eaa2ce"}
Oct 30 01:08:57.734813 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.734622    6099 scope.go:122] "RemoveContainer" containerID="371a6aa0a2fa7d8d332cb51ca58472d9965cebf3400f092c0164aa6acc98aea5"
Oct 30 01:08:57.736233 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.736204    6099 generic.go:356] "Generic (PLEG): container finished" podID="74025dac-eaf2-43b0-82d0-5470bd908c97" containerID="63354bb26bba83abd609bc2456a2f383061ae431923ea8dd348e91e375a0ed7c" exitCode=0
Oct 30 01:08:57.736345 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.736243    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp" event={"ID":"74025dac-eaf2-43b0-82d0-5470bd908c97","Type":"ContainerDied","Data":"63354bb26bba83abd609bc2456a2f383061ae431923ea8dd348e91e375a0ed7c"}
Oct 30 01:08:57.736345 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.736260    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"
Oct 30 01:08:57.736345 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.736283    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp" event={"ID":"74025dac-eaf2-43b0-82d0-5470bd908c97","Type":"ContainerDied","Data":"dfcbdb7860d9c2506485735bd70e89e424048cff615c600aee724f0e3f20c152"}
Oct 30 01:08:57.750386 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.750366    6099 scope.go:122] "RemoveContainer" containerID="afe80ec41b8c299ca906e561621eaf27a6e296ddbef21a23e8d64c5ae6c796cc"
Oct 30 01:08:57.761583 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.761568    6099 scope.go:122] "RemoveContainer" containerID="371a6aa0a2fa7d8d332cb51ca58472d9965cebf3400f092c0164aa6acc98aea5"
Oct 30 01:08:57.761881 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:08:57.761860    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"371a6aa0a2fa7d8d332cb51ca58472d9965cebf3400f092c0164aa6acc98aea5\": container with ID starting with 371a6aa0a2fa7d8d332cb51ca58472d9965cebf3400f092c0164aa6acc98aea5 not found: ID does not exist" containerID="371a6aa0a2fa7d8d332cb51ca58472d9965cebf3400f092c0164aa6acc98aea5"
Oct 30 01:08:57.761933 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.761891    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"371a6aa0a2fa7d8d332cb51ca58472d9965cebf3400f092c0164aa6acc98aea5"} err="failed to get container status \"371a6aa0a2fa7d8d332cb51ca58472d9965cebf3400f092c0164aa6acc98aea5\": rpc error: code = NotFound desc = could not find container \"371a6aa0a2fa7d8d332cb51ca58472d9965cebf3400f092c0164aa6acc98aea5\": container with ID starting with 371a6aa0a2fa7d8d332cb51ca58472d9965cebf3400f092c0164aa6acc98aea5 not found: ID does not exist"
Oct 30 01:08:57.761933 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.761908    6099 scope.go:122] "RemoveContainer" containerID="afe80ec41b8c299ca906e561621eaf27a6e296ddbef21a23e8d64c5ae6c796cc"
Oct 30 01:08:57.762167 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:08:57.762148    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe80ec41b8c299ca906e561621eaf27a6e296ddbef21a23e8d64c5ae6c796cc\": container with ID starting with afe80ec41b8c299ca906e561621eaf27a6e296ddbef21a23e8d64c5ae6c796cc not found: ID does not exist" containerID="afe80ec41b8c299ca906e561621eaf27a6e296ddbef21a23e8d64c5ae6c796cc"
Oct 30 01:08:57.762217 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.762174    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe80ec41b8c299ca906e561621eaf27a6e296ddbef21a23e8d64c5ae6c796cc"} err="failed to get container status \"afe80ec41b8c299ca906e561621eaf27a6e296ddbef21a23e8d64c5ae6c796cc\": rpc error: code = NotFound desc = could not find container \"afe80ec41b8c299ca906e561621eaf27a6e296ddbef21a23e8d64c5ae6c796cc\": container with ID starting with afe80ec41b8c299ca906e561621eaf27a6e296ddbef21a23e8d64c5ae6c796cc not found: ID does not exist"
Oct 30 01:08:57.762217 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.762191    6099 scope.go:122] "RemoveContainer" containerID="63354bb26bba83abd609bc2456a2f383061ae431923ea8dd348e91e375a0ed7c"
Oct 30 01:08:57.772144 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.772125    6099 scope.go:122] "RemoveContainer" containerID="63354bb26bba83abd609bc2456a2f383061ae431923ea8dd348e91e375a0ed7c"
Oct 30 01:08:57.772376 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:08:57.772359    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63354bb26bba83abd609bc2456a2f383061ae431923ea8dd348e91e375a0ed7c\": container with ID starting with 63354bb26bba83abd609bc2456a2f383061ae431923ea8dd348e91e375a0ed7c not found: ID does not exist" containerID="63354bb26bba83abd609bc2456a2f383061ae431923ea8dd348e91e375a0ed7c"
Oct 30 01:08:57.772433 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.772380    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63354bb26bba83abd609bc2456a2f383061ae431923ea8dd348e91e375a0ed7c"} err="failed to get container status \"63354bb26bba83abd609bc2456a2f383061ae431923ea8dd348e91e375a0ed7c\": rpc error: code = NotFound desc = could not find container \"63354bb26bba83abd609bc2456a2f383061ae431923ea8dd348e91e375a0ed7c\": container with ID starting with 63354bb26bba83abd609bc2456a2f383061ae431923ea8dd348e91e375a0ed7c not found: ID does not exist"
Oct 30 01:08:57.805854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.805823    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"]
Oct 30 01:08:57.817024 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.817003    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b3593a-openshift-default-c5596b489-2kkzw"]
Oct 30 01:08:57.846017 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.845996    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"]
Oct 30 01:08:57.849650 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:57.849630    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-hs9vp"]
Oct 30 01:08:58.263887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.263858    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c02eae9-c5d9-459d-8969-b6d3686999d3" path="/var/lib/kubelet/pods/1c02eae9-c5d9-459d-8969-b6d3686999d3/volumes"
Oct 30 01:08:58.264377 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.264363    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74025dac-eaf2-43b0-82d0-5470bd908c97" path="/var/lib/kubelet/pods/74025dac-eaf2-43b0-82d0-5470bd908c97/volumes"
Oct 30 01:08:58.864033 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.863977    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd"]
Oct 30 01:08:58.864829 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.864807    6099 cpu_manager.go:414] "containerMap: removing container" podUID="1c02eae9-c5d9-459d-8969-b6d3686999d3" containerName="istio-proxy"
Oct 30 01:08:58.864829 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.864828    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c02eae9-c5d9-459d-8969-b6d3686999d3" containerName="istio-proxy"
Oct 30 01:08:58.865037 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.864855    6099 cpu_manager.go:414] "containerMap: removing container" podUID="1c02eae9-c5d9-459d-8969-b6d3686999d3" containerName="istio-proxy"
Oct 30 01:08:58.865037 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.864862    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c02eae9-c5d9-459d-8969-b6d3686999d3" containerName="istio-proxy"
Oct 30 01:08:58.865037 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.864894    6099 cpu_manager.go:414] "containerMap: removing container" podUID="bad36020-3ebe-43b5-b83e-e996bc66e201" containerName="revoke-keys"
Oct 30 01:08:58.865037 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.864902    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad36020-3ebe-43b5-b83e-e996bc66e201" containerName="revoke-keys"
Oct 30 01:08:58.865037 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.864930    6099 cpu_manager.go:414] "containerMap: removing container" podUID="74025dac-eaf2-43b0-82d0-5470bd908c97" containerName="istio-proxy"
Oct 30 01:08:58.865037 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.864937    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74025dac-eaf2-43b0-82d0-5470bd908c97" containerName="istio-proxy"
Oct 30 01:08:58.865336 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.865127    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c02eae9-c5d9-459d-8969-b6d3686999d3" containerName="istio-proxy"
Oct 30 01:08:58.865336 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.865145    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c02eae9-c5d9-459d-8969-b6d3686999d3" containerName="istio-proxy"
Oct 30 01:08:58.865336 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.865161    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="bad36020-3ebe-43b5-b83e-e996bc66e201" containerName="revoke-keys"
Oct 30 01:08:58.865336 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.865177    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="74025dac-eaf2-43b0-82d0-5470bd908c97" containerName="istio-proxy"
Oct 30 01:08:58.870787 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.870767    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd"
Oct 30 01:08:58.885412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.885363    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:08:58.885690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.885654    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:08:58.887882 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.887861    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd"]
Oct 30 01:08:58.964206 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.964174    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlcbm\" (UniqueName: \"kubernetes.io/projected/40960901-5a4f-4584-9a78-fe0f4a8eae0a-kube-api-access-jlcbm\") pod \"maas-api-revoke-sub-d269878c-z8qpd\" (UID: \"40960901-5a4f-4584-9a78-fe0f4a8eae0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd"
Oct 30 01:08:58.964373 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.964330    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40960901-5a4f-4584-9a78-fe0f4a8eae0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d269878c-z8qpd\" (UID: \"40960901-5a4f-4584-9a78-fe0f4a8eae0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd"
Oct 30 01:08:58.964423 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:58.964406    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40960901-5a4f-4584-9a78-fe0f4a8eae0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d269878c-z8qpd\" (UID: \"40960901-5a4f-4584-9a78-fe0f4a8eae0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd"
Oct 30 01:08:59.065897 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:59.065861    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40960901-5a4f-4584-9a78-fe0f4a8eae0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d269878c-z8qpd\" (UID: \"40960901-5a4f-4584-9a78-fe0f4a8eae0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd"
Oct 30 01:08:59.066072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:59.065966    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40960901-5a4f-4584-9a78-fe0f4a8eae0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d269878c-z8qpd\" (UID: \"40960901-5a4f-4584-9a78-fe0f4a8eae0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd"
Oct 30 01:08:59.066143 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:59.066124    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlcbm\" (UniqueName: \"kubernetes.io/projected/40960901-5a4f-4584-9a78-fe0f4a8eae0a-kube-api-access-jlcbm\") pod \"maas-api-revoke-sub-d269878c-z8qpd\" (UID: \"40960901-5a4f-4584-9a78-fe0f4a8eae0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd"
Oct 30 01:08:59.066762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:59.066702    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40960901-5a4f-4584-9a78-fe0f4a8eae0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d269878c-z8qpd\" (UID: \"40960901-5a4f-4584-9a78-fe0f4a8eae0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd"
Oct 30 01:08:59.068339 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:59.068318    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40960901-5a4f-4584-9a78-fe0f4a8eae0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d269878c-z8qpd\" (UID: \"40960901-5a4f-4584-9a78-fe0f4a8eae0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd"
Oct 30 01:08:59.089318 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:59.089297    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlcbm\" (UniqueName: \"kubernetes.io/projected/40960901-5a4f-4584-9a78-fe0f4a8eae0a-kube-api-access-jlcbm\") pod \"maas-api-revoke-sub-d269878c-z8qpd\" (UID: \"40960901-5a4f-4584-9a78-fe0f4a8eae0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd"
Oct 30 01:08:59.185725 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:59.185639    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd"
Oct 30 01:08:59.489417 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:08:59.489383    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40960901_5a4f_4584_9a78_fe0f4a8eae0a.slice/crio-48929cb009f68508b729645694635f5bf6ff56a4866459a483ae9f33ceed2974 WatchSource:0}: Error finding container 48929cb009f68508b729645694635f5bf6ff56a4866459a483ae9f33ceed2974: Status 404 returned error can't find the container with id 48929cb009f68508b729645694635f5bf6ff56a4866459a483ae9f33ceed2974
Oct 30 01:08:59.490141 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:59.490117    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd"]
Oct 30 01:08:59.755254 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:08:59.755175    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd" event={"ID":"40960901-5a4f-4584-9a78-fe0f4a8eae0a","Type":"ContainerStarted","Data":"48929cb009f68508b729645694635f5bf6ff56a4866459a483ae9f33ceed2974"}
Oct 30 01:09:00.764690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:00.764605    6099 generic.go:356] "Generic (PLEG): container finished" podID="40960901-5a4f-4584-9a78-fe0f4a8eae0a" containerID="d2c3cb89bc8ba7db78deeacf854449c57a587de059cbd016f4ebe311caa17faa" exitCode=0
Oct 30 01:09:00.765106 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:00.764684    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd" event={"ID":"40960901-5a4f-4584-9a78-fe0f4a8eae0a","Type":"ContainerDied","Data":"d2c3cb89bc8ba7db78deeacf854449c57a587de059cbd016f4ebe311caa17faa"}
Oct 30 01:09:01.799432 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:01.799402    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"]
Oct 30 01:09:01.803004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:01.802983    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:09:01.817172 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:01.817144    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-6c4f6f-serving-cert\""
Oct 30 01:09:01.833393 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:01.833363    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"]
Oct 30 01:09:01.912258 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:01.912215    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48ab3ee2-9839-4203-a173-c6841b95d45a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps\" (UID: \"48ab3ee2-9839-4203-a173-c6841b95d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:09:01.912426 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:01.912266    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-575qb\" (UniqueName: \"kubernetes.io/projected/48ab3ee2-9839-4203-a173-c6841b95d45a-kube-api-access-575qb\") pod \"maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps\" (UID: \"48ab3ee2-9839-4203-a173-c6841b95d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:09:01.912426 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:01.912307    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48ab3ee2-9839-4203-a173-c6841b95d45a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps\" (UID: \"48ab3ee2-9839-4203-a173-c6841b95d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:09:01.912594 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:01.912562    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ab3ee2-9839-4203-a173-c6841b95d45a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps\" (UID: \"48ab3ee2-9839-4203-a173-c6841b95d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:09:01.995298 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:01.994943    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd"
Oct 30 01:09:02.014051 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.013976    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ab3ee2-9839-4203-a173-c6841b95d45a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps\" (UID: \"48ab3ee2-9839-4203-a173-c6841b95d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:09:02.014516 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.014494    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48ab3ee2-9839-4203-a173-c6841b95d45a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps\" (UID: \"48ab3ee2-9839-4203-a173-c6841b95d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:09:02.014647 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.014531    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-575qb\" (UniqueName: \"kubernetes.io/projected/48ab3ee2-9839-4203-a173-c6841b95d45a-kube-api-access-575qb\") pod \"maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps\" (UID: \"48ab3ee2-9839-4203-a173-c6841b95d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:09:02.014647 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.014566    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48ab3ee2-9839-4203-a173-c6841b95d45a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps\" (UID: \"48ab3ee2-9839-4203-a173-c6841b95d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:09:02.014802 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.014732    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ab3ee2-9839-4203-a173-c6841b95d45a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps\" (UID: \"48ab3ee2-9839-4203-a173-c6841b95d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:09:02.014877 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:09:02.014825    6099 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-serving-cert: secret "maas-api-e2e-shared-a-w4-6c4f6f-serving-cert" not found
Oct 30 01:09:02.014933 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:09:02.014882    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48ab3ee2-9839-4203-a173-c6841b95d45a-maas-api-tls podName:48ab3ee2-9839-4203-a173-c6841b95d45a nodeName:}" failed. No retries permitted until 2026-10-30 01:09:02.514863126 +0000 UTC m=+2036.371770862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/48ab3ee2-9839-4203-a173-c6841b95d45a-maas-api-tls") pod "maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps" (UID: "48ab3ee2-9839-4203-a173-c6841b95d45a") : secret "maas-api-e2e-shared-a-w4-6c4f6f-serving-cert" not found
Oct 30 01:09:02.017295 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.017267    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48ab3ee2-9839-4203-a173-c6841b95d45a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps\" (UID: \"48ab3ee2-9839-4203-a173-c6841b95d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:09:02.049842 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.046040    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-575qb\" (UniqueName: \"kubernetes.io/projected/48ab3ee2-9839-4203-a173-c6841b95d45a-kube-api-access-575qb\") pod \"maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps\" (UID: \"48ab3ee2-9839-4203-a173-c6841b95d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:09:02.115388 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.115351    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40960901-5a4f-4584-9a78-fe0f4a8eae0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40960901-5a4f-4584-9a78-fe0f4a8eae0a-maas-api-xks-ca\") pod \"40960901-5a4f-4584-9a78-fe0f4a8eae0a\" (UID: \"40960901-5a4f-4584-9a78-fe0f4a8eae0a\") "
Oct 30 01:09:02.115541 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.115403    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40960901-5a4f-4584-9a78-fe0f4a8eae0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40960901-5a4f-4584-9a78-fe0f4a8eae0a-maas-api-openshift-ca\") pod \"40960901-5a4f-4584-9a78-fe0f4a8eae0a\" (UID: \"40960901-5a4f-4584-9a78-fe0f4a8eae0a\") "
Oct 30 01:09:02.115676 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.115657    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40960901-5a4f-4584-9a78-fe0f4a8eae0a-kube-api-access-jlcbm\" (UniqueName: \"kubernetes.io/projected/40960901-5a4f-4584-9a78-fe0f4a8eae0a-kube-api-access-jlcbm\") pod \"40960901-5a4f-4584-9a78-fe0f4a8eae0a\" (UID: \"40960901-5a4f-4584-9a78-fe0f4a8eae0a\") "
Oct 30 01:09:02.115794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.115764    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40960901-5a4f-4584-9a78-fe0f4a8eae0a-maas-api-openshift-ca" pod "40960901-5a4f-4584-9a78-fe0f4a8eae0a" (UID: "40960901-5a4f-4584-9a78-fe0f4a8eae0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:02.116441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.116418    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40960901-5a4f-4584-9a78-fe0f4a8eae0a-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:02.117587 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.117564    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40960901-5a4f-4584-9a78-fe0f4a8eae0a-maas-api-xks-ca" pod "40960901-5a4f-4584-9a78-fe0f4a8eae0a" (UID: "40960901-5a4f-4584-9a78-fe0f4a8eae0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:02.117733 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.117692    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40960901-5a4f-4584-9a78-fe0f4a8eae0a-kube-api-access-jlcbm" pod "40960901-5a4f-4584-9a78-fe0f4a8eae0a" (UID: "40960901-5a4f-4584-9a78-fe0f4a8eae0a"). InnerVolumeSpecName "kube-api-access-jlcbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:02.218283 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.218247    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40960901-5a4f-4584-9a78-fe0f4a8eae0a-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:02.218283 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.218277    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlcbm\" (UniqueName: \"kubernetes.io/projected/40960901-5a4f-4584-9a78-fe0f4a8eae0a-kube-api-access-jlcbm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:02.394177 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.394097    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh"]
Oct 30 01:09:02.428138 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.428097    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl"]
Oct 30 01:09:02.523298 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.523254    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48ab3ee2-9839-4203-a173-c6841b95d45a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps\" (UID: \"48ab3ee2-9839-4203-a173-c6841b95d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:09:02.536744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.527090    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48ab3ee2-9839-4203-a173-c6841b95d45a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps\" (UID: \"48ab3ee2-9839-4203-a173-c6841b95d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:09:02.718616 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.718531    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:09:02.789296 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.789266    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd"
Oct 30 01:09:02.789472 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.789449    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd" event={"ID":"40960901-5a4f-4584-9a78-fe0f4a8eae0a","Type":"ContainerDied","Data":"48929cb009f68508b729645694635f5bf6ff56a4866459a483ae9f33ceed2974"}
Oct 30 01:09:02.789567 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:02.789492    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48929cb009f68508b729645694635f5bf6ff56a4866459a483ae9f33ceed2974"
Oct 30 01:09:03.038132 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:03.038041    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"]
Oct 30 01:09:03.170473 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:03.162880    6099 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:09:03.807064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:03.807006    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps" event={"ID":"48ab3ee2-9839-4203-a173-c6841b95d45a","Type":"ContainerStarted","Data":"72e26aa1957e76249b612598e2d297757212fc198ac249d2bdccde5d17ae0eb1"}
Oct 30 01:09:03.810980 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:03.810944    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps" event={"ID":"48ab3ee2-9839-4203-a173-c6841b95d45a","Type":"ContainerStarted","Data":"61d757040a761a9c5979e21b809f92937370cb1e204fb5b93f4c3d7bc23fd307"}
Oct 30 01:09:03.811174 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:03.811150    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:09:03.853345 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:03.853299    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps" podStartSLOduration=2.517816131 podStartE2EDuration="2.85328424s" totalImagesPullingTime="335.468109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:03.85190703 +0000 UTC m=+2037.708814785" watchObservedRunningTime="2026-10-30 01:09:03.85328424 +0000 UTC m=+2037.710191995"
Oct 30 01:09:07.394485 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:07.394441    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh" podUID="65fe9203-ea37-4a79-9990-5ca46ceaccd0" containerName="payload-processing" containerID="cri-o://0852f5fefd82a50ce5cfdda9ad50ad2d6cedce1091c2cfc931cf02ea74707402" gracePeriod=25
Oct 30 01:09:07.429374 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:07.429338    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl" podUID="2e720881-a65b-4e3b-95c1-e88b04a6c88a" containerName="payload-pre-processing" containerID="cri-o://d9ca08859ff4458412d7ef441e725c575c882d2654f56999ba0d21de7e9687de" gracePeriod=25
Oct 30 01:09:08.163603 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.163570    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"]
Oct 30 01:09:08.164359 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.164344    6099 cpu_manager.go:414] "containerMap: removing container" podUID="40960901-5a4f-4584-9a78-fe0f4a8eae0a" containerName="revoke-keys"
Oct 30 01:09:08.164422 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.164361    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40960901-5a4f-4584-9a78-fe0f4a8eae0a" containerName="revoke-keys"
Oct 30 01:09:08.164502 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.164492    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="40960901-5a4f-4584-9a78-fe0f4a8eae0a" containerName="revoke-keys"
Oct 30 01:09:08.167919 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.167890    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.202032 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.202004    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-16c1bb\"/\"openshift-service-ca.crt\""
Oct 30 01:09:08.202256 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.202022    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-16c1bb\"/\"kube-root-ca.crt\""
Oct 30 01:09:08.202256 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.202056    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-16c1bb\"/\"auth-test-model-16c1bb-kserve-self-signed-certs\""
Oct 30 01:09:08.202454 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.202113    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-16c1bb\"/\"default-dockercfg-pv7bm\""
Oct 30 01:09:08.206309 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.206288    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"]
Oct 30 01:09:08.338346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.338310    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-home\") pod \"auth-test-model-16c1bb-kserve-8685d9cb69-ww76s\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.338544 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.338482    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-tmp-dir\") pod \"auth-test-model-16c1bb-kserve-8685d9cb69-ww76s\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.338614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.338593    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-model-cache\") pod \"auth-test-model-16c1bb-kserve-8685d9cb69-ww76s\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.338670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.338626    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-dshm\") pod \"auth-test-model-16c1bb-kserve-8685d9cb69-ww76s\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.338738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.338691    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bbvp\" (UniqueName: \"kubernetes.io/projected/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-kube-api-access-5bbvp\") pod \"auth-test-model-16c1bb-kserve-8685d9cb69-ww76s\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.338885 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.338866    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-tls-certs\") pod \"auth-test-model-16c1bb-kserve-8685d9cb69-ww76s\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.444675 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.444441    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-tls-certs\") pod \"auth-test-model-16c1bb-kserve-8685d9cb69-ww76s\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.445349 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.445325    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-home\") pod \"auth-test-model-16c1bb-kserve-8685d9cb69-ww76s\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.445561 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.445545    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-tmp-dir\") pod \"auth-test-model-16c1bb-kserve-8685d9cb69-ww76s\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.445734 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.445720    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-model-cache\") pod \"auth-test-model-16c1bb-kserve-8685d9cb69-ww76s\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.445863 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.445849    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-dshm\") pod \"auth-test-model-16c1bb-kserve-8685d9cb69-ww76s\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.451346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.451317    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bbvp\" (UniqueName: \"kubernetes.io/projected/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-kube-api-access-5bbvp\") pod \"auth-test-model-16c1bb-kserve-8685d9cb69-ww76s\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.455674 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.452309    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-home\") pod \"auth-test-model-16c1bb-kserve-8685d9cb69-ww76s\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.456831 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.456807    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-tmp-dir\") pod \"auth-test-model-16c1bb-kserve-8685d9cb69-ww76s\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.457217 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.457190    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-model-cache\") pod \"auth-test-model-16c1bb-kserve-8685d9cb69-ww76s\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.465416 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.465377    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-dshm\") pod \"auth-test-model-16c1bb-kserve-8685d9cb69-ww76s\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.470891 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.470862    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-tls-certs\") pod \"auth-test-model-16c1bb-kserve-8685d9cb69-ww76s\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.537165 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.537131    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bbvp\" (UniqueName: \"kubernetes.io/projected/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-kube-api-access-5bbvp\") pod \"auth-test-model-16c1bb-kserve-8685d9cb69-ww76s\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") " pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:08.783477 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:08.783401    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:09.132309 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.132275    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"]
Oct 30 01:09:09.134428 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:09:09.134401    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdd0fdc4_3ea8_45ed_b05a_8f3b35bdd3a7.slice/crio-820552c76fce71fe14f9832685a636a77b42c9a10d24d7b6e795a2e37049e2bf WatchSource:0}: Error finding container 820552c76fce71fe14f9832685a636a77b42c9a10d24d7b6e795a2e37049e2bf: Status 404 returned error can't find the container with id 820552c76fce71fe14f9832685a636a77b42c9a10d24d7b6e795a2e37049e2bf
Oct 30 01:09:09.439128 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.439045    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf"]
Oct 30 01:09:09.442740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.442680    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf"
Oct 30 01:09:09.465331 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.464206    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:09:09.465331 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.464266    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:09:09.479928 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.472659    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f855a43-e9d5-4e14-9142-3a39084f2098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85089ec9-sbkgf\" (UID: \"2f855a43-e9d5-4e14-9142-3a39084f2098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf"
Oct 30 01:09:09.479928 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.472780    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f855a43-e9d5-4e14-9142-3a39084f2098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85089ec9-sbkgf\" (UID: \"2f855a43-e9d5-4e14-9142-3a39084f2098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf"
Oct 30 01:09:09.479928 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.473086    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mr6l\" (UniqueName: \"kubernetes.io/projected/2f855a43-e9d5-4e14-9142-3a39084f2098-kube-api-access-8mr6l\") pod \"maas-api-revoke-sub-85089ec9-sbkgf\" (UID: \"2f855a43-e9d5-4e14-9142-3a39084f2098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf"
Oct 30 01:09:09.479928 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.473087    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf"]
Oct 30 01:09:09.574688 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.574657    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mr6l\" (UniqueName: \"kubernetes.io/projected/2f855a43-e9d5-4e14-9142-3a39084f2098-kube-api-access-8mr6l\") pod \"maas-api-revoke-sub-85089ec9-sbkgf\" (UID: \"2f855a43-e9d5-4e14-9142-3a39084f2098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf"
Oct 30 01:09:09.574887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.574788    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f855a43-e9d5-4e14-9142-3a39084f2098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85089ec9-sbkgf\" (UID: \"2f855a43-e9d5-4e14-9142-3a39084f2098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf"
Oct 30 01:09:09.574887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.574849    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f855a43-e9d5-4e14-9142-3a39084f2098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85089ec9-sbkgf\" (UID: \"2f855a43-e9d5-4e14-9142-3a39084f2098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf"
Oct 30 01:09:09.575532 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.575509    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f855a43-e9d5-4e14-9142-3a39084f2098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85089ec9-sbkgf\" (UID: \"2f855a43-e9d5-4e14-9142-3a39084f2098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf"
Oct 30 01:09:09.577117 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.577093    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f855a43-e9d5-4e14-9142-3a39084f2098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85089ec9-sbkgf\" (UID: \"2f855a43-e9d5-4e14-9142-3a39084f2098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf"
Oct 30 01:09:09.597674 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.597646    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mr6l\" (UniqueName: \"kubernetes.io/projected/2f855a43-e9d5-4e14-9142-3a39084f2098-kube-api-access-8mr6l\") pod \"maas-api-revoke-sub-85089ec9-sbkgf\" (UID: \"2f855a43-e9d5-4e14-9142-3a39084f2098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf"
Oct 30 01:09:09.785570 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.785496    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf"
Oct 30 01:09:09.823396 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.823368    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:09:09.885363 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.885328    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s" event={"ID":"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7","Type":"ContainerStarted","Data":"35d4b24e7602bb0a5ee12a49033dbe19ad63d15782c597e486e9f2a6797dbcd8"}
Oct 30 01:09:09.885538 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.885377    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s" event={"ID":"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7","Type":"ContainerStarted","Data":"820552c76fce71fe14f9832685a636a77b42c9a10d24d7b6e795a2e37049e2bf"}
Oct 30 01:09:09.930511 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:09.930451    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s" podStartSLOduration=1.930435106 podStartE2EDuration="1.930435106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:09.930086295 +0000 UTC m=+2043.786994075" watchObservedRunningTime="2026-10-30 01:09:09.930435106 +0000 UTC m=+2043.787342863"
Oct 30 01:09:10.100135 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:09:10.100100    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f855a43_e9d5_4e14_9142_3a39084f2098.slice/crio-5b4adefe178ae54d671746610db70b089ca7d99dbbd4759bcb6488037e592627 WatchSource:0}: Error finding container 5b4adefe178ae54d671746610db70b089ca7d99dbbd4759bcb6488037e592627: Status 404 returned error can't find the container with id 5b4adefe178ae54d671746610db70b089ca7d99dbbd4759bcb6488037e592627
Oct 30 01:09:10.101451 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:10.101417    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf"]
Oct 30 01:09:10.784077 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:10.783995    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:10.896778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:10.896738    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf" event={"ID":"2f855a43-e9d5-4e14-9142-3a39084f2098","Type":"ContainerStarted","Data":"8956bb3f5adf2a646b14845bfb37d451f2db217e49b3cb7a696b064882f1864a"}
Oct 30 01:09:10.897682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:10.896811    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf" event={"ID":"2f855a43-e9d5-4e14-9142-3a39084f2098","Type":"ContainerStarted","Data":"5b4adefe178ae54d671746610db70b089ca7d99dbbd4759bcb6488037e592627"}
Oct 30 01:09:10.939854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:10.939784    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf" podStartSLOduration=1.93977004 podStartE2EDuration="1.93977004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:10.936987988 +0000 UTC m=+2044.793895747" watchObservedRunningTime="2026-10-30 01:09:10.93977004 +0000 UTC m=+2044.796677794"
Oct 30 01:09:11.905702 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:11.905669    6099 generic.go:356] "Generic (PLEG): container finished" podID="2f855a43-e9d5-4e14-9142-3a39084f2098" containerID="8956bb3f5adf2a646b14845bfb37d451f2db217e49b3cb7a696b064882f1864a" exitCode=0
Oct 30 01:09:11.906139 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:11.905742    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf" event={"ID":"2f855a43-e9d5-4e14-9142-3a39084f2098","Type":"ContainerDied","Data":"8956bb3f5adf2a646b14845bfb37d451f2db217e49b3cb7a696b064882f1864a"}
Oct 30 01:09:12.612658 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:12.612617    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh" podUID="65fe9203-ea37-4a79-9990-5ca46ceaccd0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:09:12.613437 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:12.613410    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl" podUID="2e720881-a65b-4e3b-95c1-e88b04a6c88a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:09:13.097682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:13.097656    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf"
Oct 30 01:09:13.233029 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:13.232945    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f855a43-e9d5-4e14-9142-3a39084f2098-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f855a43-e9d5-4e14-9142-3a39084f2098-maas-api-xks-ca\") pod \"2f855a43-e9d5-4e14-9142-3a39084f2098\" (UID: \"2f855a43-e9d5-4e14-9142-3a39084f2098\") "
Oct 30 01:09:13.233029 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:13.232986    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f855a43-e9d5-4e14-9142-3a39084f2098-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f855a43-e9d5-4e14-9142-3a39084f2098-maas-api-openshift-ca\") pod \"2f855a43-e9d5-4e14-9142-3a39084f2098\" (UID: \"2f855a43-e9d5-4e14-9142-3a39084f2098\") "
Oct 30 01:09:13.233235 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:13.233069    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f855a43-e9d5-4e14-9142-3a39084f2098-kube-api-access-8mr6l\" (UniqueName: \"kubernetes.io/projected/2f855a43-e9d5-4e14-9142-3a39084f2098-kube-api-access-8mr6l\") pod \"2f855a43-e9d5-4e14-9142-3a39084f2098\" (UID: \"2f855a43-e9d5-4e14-9142-3a39084f2098\") "
Oct 30 01:09:13.233372 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:13.233349    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f855a43-e9d5-4e14-9142-3a39084f2098-maas-api-openshift-ca" pod "2f855a43-e9d5-4e14-9142-3a39084f2098" (UID: "2f855a43-e9d5-4e14-9142-3a39084f2098"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:13.233732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:13.233718    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f855a43-e9d5-4e14-9142-3a39084f2098-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:13.235133 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:13.235113    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f855a43-e9d5-4e14-9142-3a39084f2098-maas-api-xks-ca" pod "2f855a43-e9d5-4e14-9142-3a39084f2098" (UID: "2f855a43-e9d5-4e14-9142-3a39084f2098"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:13.235241 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:13.235170    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f855a43-e9d5-4e14-9142-3a39084f2098-kube-api-access-8mr6l" pod "2f855a43-e9d5-4e14-9142-3a39084f2098" (UID: "2f855a43-e9d5-4e14-9142-3a39084f2098"). InnerVolumeSpecName "kube-api-access-8mr6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:13.335296 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:13.335264    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f855a43-e9d5-4e14-9142-3a39084f2098-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:13.335296 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:13.335291    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mr6l\" (UniqueName: \"kubernetes.io/projected/2f855a43-e9d5-4e14-9142-3a39084f2098-kube-api-access-8mr6l\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:13.922919 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:13.922880    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf"
Oct 30 01:09:13.922919 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:13.922907    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf" event={"ID":"2f855a43-e9d5-4e14-9142-3a39084f2098","Type":"ContainerDied","Data":"5b4adefe178ae54d671746610db70b089ca7d99dbbd4759bcb6488037e592627"}
Oct 30 01:09:13.923168 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:13.922965    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b4adefe178ae54d671746610db70b089ca7d99dbbd4759bcb6488037e592627"
Oct 30 01:09:16.630701 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:09:16.630662    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:09:18.783795 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:18.783762    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:18.796498 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:18.796474    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:18.975160 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:18.975129    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:09:19.963983 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:19.963951    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq"]
Oct 30 01:09:19.971739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:19.971695    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49dfaf-f5hnq"]
Oct 30 01:09:20.263894 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:20.263814    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="160b4ab9-ba38-4572-bcec-c89aebb40117" path="/var/lib/kubelet/pods/160b4ab9-ba38-4572-bcec-c89aebb40117/volumes"
Oct 30 01:09:21.575088 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:21.575054    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"]
Oct 30 01:09:21.575602 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:21.575282    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7" podUID="70cef62b-b6ff-4aca-9ba3-680d50316121" containerName="maas-api" containerID="cri-o://3f4a824958a1aba34f129c415f620341c7b5f216257dfee78fca068bd919722d" gracePeriod=30
Oct 30 01:09:21.964951 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:21.964928    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"
Oct 30 01:09:22.006006 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.005972    6099 generic.go:356] "Generic (PLEG): container finished" podID="70cef62b-b6ff-4aca-9ba3-680d50316121" containerID="3f4a824958a1aba34f129c415f620341c7b5f216257dfee78fca068bd919722d" exitCode=0
Oct 30 01:09:22.006219 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.006078    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7" event={"ID":"70cef62b-b6ff-4aca-9ba3-680d50316121","Type":"ContainerDied","Data":"3f4a824958a1aba34f129c415f620341c7b5f216257dfee78fca068bd919722d"}
Oct 30 01:09:22.006219 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.006130    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7" event={"ID":"70cef62b-b6ff-4aca-9ba3-680d50316121","Type":"ContainerDied","Data":"f99ebb064fc41a3cf19746e85136ef81ae96d8d2cc69cd7c9d371aaca802e42f"}
Oct 30 01:09:22.006219 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.006159    6099 scope.go:122] "RemoveContainer" containerID="3f4a824958a1aba34f129c415f620341c7b5f216257dfee78fca068bd919722d"
Oct 30 01:09:22.006532 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.006511    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"
Oct 30 01:09:22.035426 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.035396    6099 scope.go:122] "RemoveContainer" containerID="3f4a824958a1aba34f129c415f620341c7b5f216257dfee78fca068bd919722d"
Oct 30 01:09:22.035802 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:09:22.035767    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f4a824958a1aba34f129c415f620341c7b5f216257dfee78fca068bd919722d\": container with ID starting with 3f4a824958a1aba34f129c415f620341c7b5f216257dfee78fca068bd919722d not found: ID does not exist" containerID="3f4a824958a1aba34f129c415f620341c7b5f216257dfee78fca068bd919722d"
Oct 30 01:09:22.035958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.035809    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f4a824958a1aba34f129c415f620341c7b5f216257dfee78fca068bd919722d"} err="failed to get container status \"3f4a824958a1aba34f129c415f620341c7b5f216257dfee78fca068bd919722d\": rpc error: code = NotFound desc = could not find container \"3f4a824958a1aba34f129c415f620341c7b5f216257dfee78fca068bd919722d\": container with ID starting with 3f4a824958a1aba34f129c415f620341c7b5f216257dfee78fca068bd919722d not found: ID does not exist"
Oct 30 01:09:22.127809 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.127776    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70cef62b-b6ff-4aca-9ba3-680d50316121-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70cef62b-b6ff-4aca-9ba3-680d50316121-maas-api-tls\") pod \"70cef62b-b6ff-4aca-9ba3-680d50316121\" (UID: \"70cef62b-b6ff-4aca-9ba3-680d50316121\") "
Oct 30 01:09:22.128014 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.127980    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70cef62b-b6ff-4aca-9ba3-680d50316121-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70cef62b-b6ff-4aca-9ba3-680d50316121-metrics-certs\") pod \"70cef62b-b6ff-4aca-9ba3-680d50316121\" (UID: \"70cef62b-b6ff-4aca-9ba3-680d50316121\") "
Oct 30 01:09:22.128151 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.128126    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70cef62b-b6ff-4aca-9ba3-680d50316121-kube-api-access-lvq5x\" (UniqueName: \"kubernetes.io/projected/70cef62b-b6ff-4aca-9ba3-680d50316121-kube-api-access-lvq5x\") pod \"70cef62b-b6ff-4aca-9ba3-680d50316121\" (UID: \"70cef62b-b6ff-4aca-9ba3-680d50316121\") "
Oct 30 01:09:22.128244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.128215    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70cef62b-b6ff-4aca-9ba3-680d50316121-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70cef62b-b6ff-4aca-9ba3-680d50316121-openshift-service-ca\") pod \"70cef62b-b6ff-4aca-9ba3-680d50316121\" (UID: \"70cef62b-b6ff-4aca-9ba3-680d50316121\") "
Oct 30 01:09:22.129448 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.129420    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70cef62b-b6ff-4aca-9ba3-680d50316121-openshift-service-ca" pod "70cef62b-b6ff-4aca-9ba3-680d50316121" (UID: "70cef62b-b6ff-4aca-9ba3-680d50316121"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:22.131137 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.131106    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70cef62b-b6ff-4aca-9ba3-680d50316121-kube-api-access-lvq5x" pod "70cef62b-b6ff-4aca-9ba3-680d50316121" (UID: "70cef62b-b6ff-4aca-9ba3-680d50316121"). InnerVolumeSpecName "kube-api-access-lvq5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:22.131310 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.131237    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70cef62b-b6ff-4aca-9ba3-680d50316121-metrics-certs" pod "70cef62b-b6ff-4aca-9ba3-680d50316121" (UID: "70cef62b-b6ff-4aca-9ba3-680d50316121"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:22.155122 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.155031    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70cef62b-b6ff-4aca-9ba3-680d50316121-maas-api-tls" pod "70cef62b-b6ff-4aca-9ba3-680d50316121" (UID: "70cef62b-b6ff-4aca-9ba3-680d50316121"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:22.230816 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.230782    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvq5x\" (UniqueName: \"kubernetes.io/projected/70cef62b-b6ff-4aca-9ba3-680d50316121-kube-api-access-lvq5x\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:22.230816 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.230811    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70cef62b-b6ff-4aca-9ba3-680d50316121-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:22.230816 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.230822    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70cef62b-b6ff-4aca-9ba3-680d50316121-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:22.231081 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.230836    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70cef62b-b6ff-4aca-9ba3-680d50316121-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:22.348558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.348524    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"]
Oct 30 01:09:22.360170 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.360138    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd257f-86696c9494-qqxk7"]
Oct 30 01:09:22.502240 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.502148    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx"]
Oct 30 01:09:22.506602 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.506572    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd257f-v8zgx"]
Oct 30 01:09:22.613236 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.613188    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl" podUID="2e720881-a65b-4e3b-95c1-e88b04a6c88a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:09:22.613715 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:22.613466    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh" podUID="65fe9203-ea37-4a79-9990-5ca46ceaccd0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:09:24.264272 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:24.264218    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70cef62b-b6ff-4aca-9ba3-680d50316121" path="/var/lib/kubelet/pods/70cef62b-b6ff-4aca-9ba3-680d50316121/volumes"
Oct 30 01:09:24.264763 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:24.264744    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87dbd409-a029-4bcb-bf29-22fba38e0c24" path="/var/lib/kubelet/pods/87dbd409-a029-4bcb-bf29-22fba38e0c24/volumes"
Oct 30 01:09:26.152793 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.152757    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76"]
Oct 30 01:09:26.153524 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.153508    6099 cpu_manager.go:414] "containerMap: removing container" podUID="70cef62b-b6ff-4aca-9ba3-680d50316121" containerName="maas-api"
Oct 30 01:09:26.153570 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.153525    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70cef62b-b6ff-4aca-9ba3-680d50316121" containerName="maas-api"
Oct 30 01:09:26.153612 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.153581    6099 cpu_manager.go:414] "containerMap: removing container" podUID="2f855a43-e9d5-4e14-9142-3a39084f2098" containerName="revoke-keys"
Oct 30 01:09:26.153612 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.153588    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f855a43-e9d5-4e14-9142-3a39084f2098" containerName="revoke-keys"
Oct 30 01:09:26.153739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.153724    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f855a43-e9d5-4e14-9142-3a39084f2098" containerName="revoke-keys"
Oct 30 01:09:26.153785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.153742    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="70cef62b-b6ff-4aca-9ba3-680d50316121" containerName="maas-api"
Oct 30 01:09:26.159050 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.159033    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76"
Oct 30 01:09:26.176201 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.176179    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76"]
Oct 30 01:09:26.187726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.184366    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:09:26.187726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.184460    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:09:26.205270 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.205240    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnrtf\" (UniqueName: \"kubernetes.io/projected/e0638e9c-11e9-4aa5-b98d-15b8760b6399-kube-api-access-fnrtf\") pod \"maas-api-revoke-sub-a7ba0956-vmq76\" (UID: \"e0638e9c-11e9-4aa5-b98d-15b8760b6399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76"
Oct 30 01:09:26.205405 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.205334    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0638e9c-11e9-4aa5-b98d-15b8760b6399-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7ba0956-vmq76\" (UID: \"e0638e9c-11e9-4aa5-b98d-15b8760b6399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76"
Oct 30 01:09:26.205405 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.205366    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0638e9c-11e9-4aa5-b98d-15b8760b6399-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7ba0956-vmq76\" (UID: \"e0638e9c-11e9-4aa5-b98d-15b8760b6399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76"
Oct 30 01:09:26.307148 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.307112    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0638e9c-11e9-4aa5-b98d-15b8760b6399-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7ba0956-vmq76\" (UID: \"e0638e9c-11e9-4aa5-b98d-15b8760b6399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76"
Oct 30 01:09:26.307148 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.307151    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0638e9c-11e9-4aa5-b98d-15b8760b6399-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7ba0956-vmq76\" (UID: \"e0638e9c-11e9-4aa5-b98d-15b8760b6399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76"
Oct 30 01:09:26.307749 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.307681    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnrtf\" (UniqueName: \"kubernetes.io/projected/e0638e9c-11e9-4aa5-b98d-15b8760b6399-kube-api-access-fnrtf\") pod \"maas-api-revoke-sub-a7ba0956-vmq76\" (UID: \"e0638e9c-11e9-4aa5-b98d-15b8760b6399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76"
Oct 30 01:09:26.307937 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.307915    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0638e9c-11e9-4aa5-b98d-15b8760b6399-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7ba0956-vmq76\" (UID: \"e0638e9c-11e9-4aa5-b98d-15b8760b6399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76"
Oct 30 01:09:26.310042 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.310017    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0638e9c-11e9-4aa5-b98d-15b8760b6399-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7ba0956-vmq76\" (UID: \"e0638e9c-11e9-4aa5-b98d-15b8760b6399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76"
Oct 30 01:09:26.330680 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.330653    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnrtf\" (UniqueName: \"kubernetes.io/projected/e0638e9c-11e9-4aa5-b98d-15b8760b6399-kube-api-access-fnrtf\") pod \"maas-api-revoke-sub-a7ba0956-vmq76\" (UID: \"e0638e9c-11e9-4aa5-b98d-15b8760b6399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76"
Oct 30 01:09:26.473189 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.473111    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76"
Oct 30 01:09:26.783728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:26.783307    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76"]
Oct 30 01:09:26.786061 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:09:26.786033    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0638e9c_11e9_4aa5_b98d_15b8760b6399.slice/crio-785b3874a8dbcb567f563411e25eb669854a84cab61d6d47f8a1a5a77b885506 WatchSource:0}: Error finding container 785b3874a8dbcb567f563411e25eb669854a84cab61d6d47f8a1a5a77b885506: Status 404 returned error can't find the container with id 785b3874a8dbcb567f563411e25eb669854a84cab61d6d47f8a1a5a77b885506
Oct 30 01:09:27.061694 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.061658    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76" event={"ID":"e0638e9c-11e9-4aa5-b98d-15b8760b6399","Type":"ContainerStarted","Data":"785b3874a8dbcb567f563411e25eb669854a84cab61d6d47f8a1a5a77b885506"}
Oct 30 01:09:27.242755 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.241642    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"]
Oct 30 01:09:27.242755 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.241959    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj" podUID="f66406e6-2198-443d-8130-31b590e4f566" containerName="istio-proxy" containerID="cri-o://c81ee7acbcbadc3ac360ad5f91973475a9313d8d30ba27bc037910e69b8a692c" gracePeriod=30
Oct 30 01:09:27.750889 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.750853    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr"]
Oct 30 01:09:27.755700 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.755679    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr"
Oct 30 01:09:27.763297 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.763276    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh"
Oct 30 01:09:27.775998 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.775898    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr"]
Oct 30 01:09:27.795549 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.795526    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl"
Oct 30 01:09:27.838423 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.838396    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65fe9203-ea37-4a79-9990-5ca46ceaccd0-kube-api-access-fk5zj\" (UniqueName: \"kubernetes.io/projected/65fe9203-ea37-4a79-9990-5ca46ceaccd0-kube-api-access-fk5zj\") pod \"65fe9203-ea37-4a79-9990-5ca46ceaccd0\" (UID: \"65fe9203-ea37-4a79-9990-5ca46ceaccd0\") "
Oct 30 01:09:27.838726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.838690    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e720881-a65b-4e3b-95c1-e88b04a6c88a-kube-api-access-lnb7k\" (UniqueName: \"kubernetes.io/projected/2e720881-a65b-4e3b-95c1-e88b04a6c88a-kube-api-access-lnb7k\") pod \"2e720881-a65b-4e3b-95c1-e88b04a6c88a\" (UID: \"2e720881-a65b-4e3b-95c1-e88b04a6c88a\") "
Oct 30 01:09:27.838950 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.838924    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e720881-a65b-4e3b-95c1-e88b04a6c88a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e720881-a65b-4e3b-95c1-e88b04a6c88a-plugins-config-volume\") pod \"2e720881-a65b-4e3b-95c1-e88b04a6c88a\" (UID: \"2e720881-a65b-4e3b-95c1-e88b04a6c88a\") "
Oct 30 01:09:27.839146 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.838963    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65fe9203-ea37-4a79-9990-5ca46ceaccd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65fe9203-ea37-4a79-9990-5ca46ceaccd0-plugins-config-volume\") pod \"65fe9203-ea37-4a79-9990-5ca46ceaccd0\" (UID: \"65fe9203-ea37-4a79-9990-5ca46ceaccd0\") "
Oct 30 01:09:27.839349 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.839328    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e720881-a65b-4e3b-95c1-e88b04a6c88a-plugins-config-volume" pod "2e720881-a65b-4e3b-95c1-e88b04a6c88a" (UID: "2e720881-a65b-4e3b-95c1-e88b04a6c88a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:27.839419 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.839343    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928d6088-b7d7-4052-89eb-9abba7b186a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr\" (UID: \"928d6088-b7d7-4052-89eb-9abba7b186a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr"
Oct 30 01:09:27.840854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.839545    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65fe9203-ea37-4a79-9990-5ca46ceaccd0-plugins-config-volume" pod "65fe9203-ea37-4a79-9990-5ca46ceaccd0" (UID: "65fe9203-ea37-4a79-9990-5ca46ceaccd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:27.840854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.839722    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928d6088-b7d7-4052-89eb-9abba7b186a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr\" (UID: \"928d6088-b7d7-4052-89eb-9abba7b186a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr"
Oct 30 01:09:27.840854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.840134    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srmbk\" (UniqueName: \"kubernetes.io/projected/928d6088-b7d7-4052-89eb-9abba7b186a0-kube-api-access-srmbk\") pod \"maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr\" (UID: \"928d6088-b7d7-4052-89eb-9abba7b186a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr"
Oct 30 01:09:27.840854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.840356    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e720881-a65b-4e3b-95c1-e88b04a6c88a-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:27.840854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.840378    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65fe9203-ea37-4a79-9990-5ca46ceaccd0-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:27.841920 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.841893    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65fe9203-ea37-4a79-9990-5ca46ceaccd0-kube-api-access-fk5zj" pod "65fe9203-ea37-4a79-9990-5ca46ceaccd0" (UID: "65fe9203-ea37-4a79-9990-5ca46ceaccd0"). InnerVolumeSpecName "kube-api-access-fk5zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:27.842031 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.841985    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e720881-a65b-4e3b-95c1-e88b04a6c88a-kube-api-access-lnb7k" pod "2e720881-a65b-4e3b-95c1-e88b04a6c88a" (UID: "2e720881-a65b-4e3b-95c1-e88b04a6c88a"). InnerVolumeSpecName "kube-api-access-lnb7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:27.941810 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.941775    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928d6088-b7d7-4052-89eb-9abba7b186a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr\" (UID: \"928d6088-b7d7-4052-89eb-9abba7b186a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr"
Oct 30 01:09:27.941964 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.941905    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srmbk\" (UniqueName: \"kubernetes.io/projected/928d6088-b7d7-4052-89eb-9abba7b186a0-kube-api-access-srmbk\") pod \"maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr\" (UID: \"928d6088-b7d7-4052-89eb-9abba7b186a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr"
Oct 30 01:09:27.941964 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.941944    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928d6088-b7d7-4052-89eb-9abba7b186a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr\" (UID: \"928d6088-b7d7-4052-89eb-9abba7b186a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr"
Oct 30 01:09:27.942087 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.942075    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk5zj\" (UniqueName: \"kubernetes.io/projected/65fe9203-ea37-4a79-9990-5ca46ceaccd0-kube-api-access-fk5zj\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:27.942149 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.942089    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnb7k\" (UniqueName: \"kubernetes.io/projected/2e720881-a65b-4e3b-95c1-e88b04a6c88a-kube-api-access-lnb7k\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:27.942322 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.942302    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928d6088-b7d7-4052-89eb-9abba7b186a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr\" (UID: \"928d6088-b7d7-4052-89eb-9abba7b186a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr"
Oct 30 01:09:27.944170 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.944144    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928d6088-b7d7-4052-89eb-9abba7b186a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr\" (UID: \"928d6088-b7d7-4052-89eb-9abba7b186a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr"
Oct 30 01:09:27.964545 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:27.964517    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srmbk\" (UniqueName: \"kubernetes.io/projected/928d6088-b7d7-4052-89eb-9abba7b186a0-kube-api-access-srmbk\") pod \"maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr\" (UID: \"928d6088-b7d7-4052-89eb-9abba7b186a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr"
Oct 30 01:09:28.070892 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.070855    6099 generic.go:356] "Generic (PLEG): container finished" podID="2e720881-a65b-4e3b-95c1-e88b04a6c88a" containerID="d9ca08859ff4458412d7ef441e725c575c882d2654f56999ba0d21de7e9687de" exitCode=0
Oct 30 01:09:28.071039 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.070949    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl" event={"ID":"2e720881-a65b-4e3b-95c1-e88b04a6c88a","Type":"ContainerDied","Data":"d9ca08859ff4458412d7ef441e725c575c882d2654f56999ba0d21de7e9687de"}
Oct 30 01:09:28.071039 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.070996    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl" event={"ID":"2e720881-a65b-4e3b-95c1-e88b04a6c88a","Type":"ContainerDied","Data":"6d87accc6cf5317b782681b765f90a8cfbc363b129bb514585914f8d417c6cc1"}
Oct 30 01:09:28.071039 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.070954    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl"
Oct 30 01:09:28.071213 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.071019    6099 scope.go:122] "RemoveContainer" containerID="d9ca08859ff4458412d7ef441e725c575c882d2654f56999ba0d21de7e9687de"
Oct 30 01:09:28.072775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.072749    6099 generic.go:356] "Generic (PLEG): container finished" podID="e0638e9c-11e9-4aa5-b98d-15b8760b6399" containerID="9cf327e31878fe32c1bce027595620d3304bd91b3b75182b01270fc5b6d77234" exitCode=0
Oct 30 01:09:28.072902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.072837    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76" event={"ID":"e0638e9c-11e9-4aa5-b98d-15b8760b6399","Type":"ContainerDied","Data":"9cf327e31878fe32c1bce027595620d3304bd91b3b75182b01270fc5b6d77234"}
Oct 30 01:09:28.074368 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.074348    6099 generic.go:356] "Generic (PLEG): container finished" podID="65fe9203-ea37-4a79-9990-5ca46ceaccd0" containerID="0852f5fefd82a50ce5cfdda9ad50ad2d6cedce1091c2cfc931cf02ea74707402" exitCode=0
Oct 30 01:09:28.074478 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.074403    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh"
Oct 30 01:09:28.074478 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.074429    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh" event={"ID":"65fe9203-ea37-4a79-9990-5ca46ceaccd0","Type":"ContainerDied","Data":"0852f5fefd82a50ce5cfdda9ad50ad2d6cedce1091c2cfc931cf02ea74707402"}
Oct 30 01:09:28.074478 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.074466    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh" event={"ID":"65fe9203-ea37-4a79-9990-5ca46ceaccd0","Type":"ContainerDied","Data":"67ac7603bff864aad86429595e5249d08e7b35684b2976d8a7fb01bcb3195239"}
Oct 30 01:09:28.082893 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.082874    6099 scope.go:122] "RemoveContainer" containerID="d9ca08859ff4458412d7ef441e725c575c882d2654f56999ba0d21de7e9687de"
Oct 30 01:09:28.083212 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:09:28.083193    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ca08859ff4458412d7ef441e725c575c882d2654f56999ba0d21de7e9687de\": container with ID starting with d9ca08859ff4458412d7ef441e725c575c882d2654f56999ba0d21de7e9687de not found: ID does not exist" containerID="d9ca08859ff4458412d7ef441e725c575c882d2654f56999ba0d21de7e9687de"
Oct 30 01:09:28.083296 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.083225    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ca08859ff4458412d7ef441e725c575c882d2654f56999ba0d21de7e9687de"} err="failed to get container status \"d9ca08859ff4458412d7ef441e725c575c882d2654f56999ba0d21de7e9687de\": rpc error: code = NotFound desc = could not find container \"d9ca08859ff4458412d7ef441e725c575c882d2654f56999ba0d21de7e9687de\": container with ID starting with d9ca08859ff4458412d7ef441e725c575c882d2654f56999ba0d21de7e9687de not found: ID does not exist"
Oct 30 01:09:28.083296 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.083241    6099 scope.go:122] "RemoveContainer" containerID="0852f5fefd82a50ce5cfdda9ad50ad2d6cedce1091c2cfc931cf02ea74707402"
Oct 30 01:09:28.093333 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.093313    6099 scope.go:122] "RemoveContainer" containerID="0852f5fefd82a50ce5cfdda9ad50ad2d6cedce1091c2cfc931cf02ea74707402"
Oct 30 01:09:28.093507 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.093491    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr"
Oct 30 01:09:28.093601 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:09:28.093578    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0852f5fefd82a50ce5cfdda9ad50ad2d6cedce1091c2cfc931cf02ea74707402\": container with ID starting with 0852f5fefd82a50ce5cfdda9ad50ad2d6cedce1091c2cfc931cf02ea74707402 not found: ID does not exist" containerID="0852f5fefd82a50ce5cfdda9ad50ad2d6cedce1091c2cfc931cf02ea74707402"
Oct 30 01:09:28.093649 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.093609    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0852f5fefd82a50ce5cfdda9ad50ad2d6cedce1091c2cfc931cf02ea74707402"} err="failed to get container status \"0852f5fefd82a50ce5cfdda9ad50ad2d6cedce1091c2cfc931cf02ea74707402\": rpc error: code = NotFound desc = could not find container \"0852f5fefd82a50ce5cfdda9ad50ad2d6cedce1091c2cfc931cf02ea74707402\": container with ID starting with 0852f5fefd82a50ce5cfdda9ad50ad2d6cedce1091c2cfc931cf02ea74707402 not found: ID does not exist"
Oct 30 01:09:28.138089 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.138029    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh"]
Oct 30 01:09:28.142640 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.142612    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-780077-667bdfc946-hdlwh"]
Oct 30 01:09:28.244295 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.244257    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl"]
Oct 30 01:09:28.249383 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.249353    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-780077-8fb65bb75-dr7jl"]
Oct 30 01:09:28.275979 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.275940    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e720881-a65b-4e3b-95c1-e88b04a6c88a" path="/var/lib/kubelet/pods/2e720881-a65b-4e3b-95c1-e88b04a6c88a/volumes"
Oct 30 01:09:28.276676 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.276653    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65fe9203-ea37-4a79-9990-5ca46ceaccd0" path="/var/lib/kubelet/pods/65fe9203-ea37-4a79-9990-5ca46ceaccd0/volumes"
Oct 30 01:09:28.401030 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:09:28.401004    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928d6088_b7d7_4052_89eb_9abba7b186a0.slice/crio-8bccfba1d0ad3db738043218d28e242f43c14db06fdeb2da7cb7c28dbc9aa789 WatchSource:0}: Error finding container 8bccfba1d0ad3db738043218d28e242f43c14db06fdeb2da7cb7c28dbc9aa789: Status 404 returned error can't find the container with id 8bccfba1d0ad3db738043218d28e242f43c14db06fdeb2da7cb7c28dbc9aa789
Oct 30 01:09:28.403585 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.403559    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr"]
Oct 30 01:09:28.758654 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.758625    6099 patch_prober.go:29] interesting pod/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:09:28.758814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:28.758673    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj" podUID="f66406e6-2198-443d-8130-31b590e4f566" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:09:29.084027 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:29.083988    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr" event={"ID":"928d6088-b7d7-4052-89eb-9abba7b186a0","Type":"ContainerStarted","Data":"c703ef40d58c2f153f161fb98aa981ad26664f5ac11846954bf154e55ae1f913"}
Oct 30 01:09:29.084237 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:29.084044    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr" event={"ID":"928d6088-b7d7-4052-89eb-9abba7b186a0","Type":"ContainerStarted","Data":"8bccfba1d0ad3db738043218d28e242f43c14db06fdeb2da7cb7c28dbc9aa789"}
Oct 30 01:09:29.133906 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:29.133851    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr" podStartSLOduration=2.133835087 podStartE2EDuration="2.133835087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:29.131849019 +0000 UTC m=+2062.988756774" watchObservedRunningTime="2026-10-30 01:09:29.133835087 +0000 UTC m=+2062.990742842"
Oct 30 01:09:29.267214 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:29.267189    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76"
Oct 30 01:09:29.369466 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:29.369378    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0638e9c-11e9-4aa5-b98d-15b8760b6399-kube-api-access-fnrtf\" (UniqueName: \"kubernetes.io/projected/e0638e9c-11e9-4aa5-b98d-15b8760b6399-kube-api-access-fnrtf\") pod \"e0638e9c-11e9-4aa5-b98d-15b8760b6399\" (UID: \"e0638e9c-11e9-4aa5-b98d-15b8760b6399\") "
Oct 30 01:09:29.369627 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:29.369574    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0638e9c-11e9-4aa5-b98d-15b8760b6399-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0638e9c-11e9-4aa5-b98d-15b8760b6399-maas-api-openshift-ca\") pod \"e0638e9c-11e9-4aa5-b98d-15b8760b6399\" (UID: \"e0638e9c-11e9-4aa5-b98d-15b8760b6399\") "
Oct 30 01:09:29.369922 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:29.369896    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0638e9c-11e9-4aa5-b98d-15b8760b6399-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0638e9c-11e9-4aa5-b98d-15b8760b6399-maas-api-xks-ca\") pod \"e0638e9c-11e9-4aa5-b98d-15b8760b6399\" (UID: \"e0638e9c-11e9-4aa5-b98d-15b8760b6399\") "
Oct 30 01:09:29.371217 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:29.371184    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0638e9c-11e9-4aa5-b98d-15b8760b6399-maas-api-openshift-ca" pod "e0638e9c-11e9-4aa5-b98d-15b8760b6399" (UID: "e0638e9c-11e9-4aa5-b98d-15b8760b6399"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:29.379764 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:29.374695    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0638e9c-11e9-4aa5-b98d-15b8760b6399-maas-api-xks-ca" pod "e0638e9c-11e9-4aa5-b98d-15b8760b6399" (UID: "e0638e9c-11e9-4aa5-b98d-15b8760b6399"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:29.379764 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:29.374775    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0638e9c-11e9-4aa5-b98d-15b8760b6399-kube-api-access-fnrtf" pod "e0638e9c-11e9-4aa5-b98d-15b8760b6399" (UID: "e0638e9c-11e9-4aa5-b98d-15b8760b6399"). InnerVolumeSpecName "kube-api-access-fnrtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:29.478493 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:29.478392    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0638e9c-11e9-4aa5-b98d-15b8760b6399-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:29.478493 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:29.478425    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnrtf\" (UniqueName: \"kubernetes.io/projected/e0638e9c-11e9-4aa5-b98d-15b8760b6399-kube-api-access-fnrtf\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:29.478797 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:29.478441    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0638e9c-11e9-4aa5-b98d-15b8760b6399-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:30.098904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:30.098865    6099 generic.go:356] "Generic (PLEG): container finished" podID="928d6088-b7d7-4052-89eb-9abba7b186a0" containerID="c703ef40d58c2f153f161fb98aa981ad26664f5ac11846954bf154e55ae1f913" exitCode=0
Oct 30 01:09:30.099117 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:30.098961    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr" event={"ID":"928d6088-b7d7-4052-89eb-9abba7b186a0","Type":"ContainerDied","Data":"c703ef40d58c2f153f161fb98aa981ad26664f5ac11846954bf154e55ae1f913"}
Oct 30 01:09:30.105205 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:30.100953    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76"
Oct 30 01:09:30.105205 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:30.100978    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76" event={"ID":"e0638e9c-11e9-4aa5-b98d-15b8760b6399","Type":"ContainerDied","Data":"785b3874a8dbcb567f563411e25eb669854a84cab61d6d47f8a1a5a77b885506"}
Oct 30 01:09:30.105205 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:30.101115    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="785b3874a8dbcb567f563411e25eb669854a84cab61d6d47f8a1a5a77b885506"
Oct 30 01:09:31.320023 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:31.319994    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr"
Oct 30 01:09:31.425056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:31.422723    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/928d6088-b7d7-4052-89eb-9abba7b186a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928d6088-b7d7-4052-89eb-9abba7b186a0-maas-api-openshift-ca\") pod \"928d6088-b7d7-4052-89eb-9abba7b186a0\" (UID: \"928d6088-b7d7-4052-89eb-9abba7b186a0\") "
Oct 30 01:09:31.425056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:31.422958    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/928d6088-b7d7-4052-89eb-9abba7b186a0-kube-api-access-srmbk\" (UniqueName: \"kubernetes.io/projected/928d6088-b7d7-4052-89eb-9abba7b186a0-kube-api-access-srmbk\") pod \"928d6088-b7d7-4052-89eb-9abba7b186a0\" (UID: \"928d6088-b7d7-4052-89eb-9abba7b186a0\") "
Oct 30 01:09:31.425056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:31.423051    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/928d6088-b7d7-4052-89eb-9abba7b186a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928d6088-b7d7-4052-89eb-9abba7b186a0-maas-api-xks-ca\") pod \"928d6088-b7d7-4052-89eb-9abba7b186a0\" (UID: \"928d6088-b7d7-4052-89eb-9abba7b186a0\") "
Oct 30 01:09:31.425056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:31.423145    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/928d6088-b7d7-4052-89eb-9abba7b186a0-maas-api-openshift-ca" pod "928d6088-b7d7-4052-89eb-9abba7b186a0" (UID: "928d6088-b7d7-4052-89eb-9abba7b186a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:31.425056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:31.423982    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928d6088-b7d7-4052-89eb-9abba7b186a0-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:31.425488 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:31.425329    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928d6088-b7d7-4052-89eb-9abba7b186a0-kube-api-access-srmbk" pod "928d6088-b7d7-4052-89eb-9abba7b186a0" (UID: "928d6088-b7d7-4052-89eb-9abba7b186a0"). InnerVolumeSpecName "kube-api-access-srmbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:31.428730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:31.425811    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/928d6088-b7d7-4052-89eb-9abba7b186a0-maas-api-xks-ca" pod "928d6088-b7d7-4052-89eb-9abba7b186a0" (UID: "928d6088-b7d7-4052-89eb-9abba7b186a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:31.525535 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:31.525499    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srmbk\" (UniqueName: \"kubernetes.io/projected/928d6088-b7d7-4052-89eb-9abba7b186a0-kube-api-access-srmbk\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:31.525535 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:31.525534    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928d6088-b7d7-4052-89eb-9abba7b186a0-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:32.120379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.120350    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr"
Oct 30 01:09:32.120583 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.120349    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr" event={"ID":"928d6088-b7d7-4052-89eb-9abba7b186a0","Type":"ContainerDied","Data":"8bccfba1d0ad3db738043218d28e242f43c14db06fdeb2da7cb7c28dbc9aa789"}
Oct 30 01:09:32.120583 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.120502    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bccfba1d0ad3db738043218d28e242f43c14db06fdeb2da7cb7c28dbc9aa789"
Oct 30 01:09:32.528261 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.528231    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:09:32.643345 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.643308    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-credential-socket\") pod \"f66406e6-2198-443d-8130-31b590e4f566\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") "
Oct 30 01:09:32.643540 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.643406    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-workload-socket\") pod \"f66406e6-2198-443d-8130-31b590e4f566\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") "
Oct 30 01:09:32.643540 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.643439    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-istio-envoy\") pod \"f66406e6-2198-443d-8130-31b590e4f566\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") "
Oct 30 01:09:32.643679 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.643547    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66406e6-2198-443d-8130-31b590e4f566-istio-token\" (UniqueName: \"kubernetes.io/projected/f66406e6-2198-443d-8130-31b590e4f566-istio-token\") pod \"f66406e6-2198-443d-8130-31b590e4f566\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") "
Oct 30 01:09:32.643679 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.643577    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f66406e6-2198-443d-8130-31b590e4f566-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f66406e6-2198-443d-8130-31b590e4f566-istio-podinfo\") pod \"f66406e6-2198-443d-8130-31b590e4f566\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") "
Oct 30 01:09:32.643679 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.643637    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-credential-socket" pod "f66406e6-2198-443d-8130-31b590e4f566" (UID: "f66406e6-2198-443d-8130-31b590e4f566"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:32.643904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.643732    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66406e6-2198-443d-8130-31b590e4f566-kube-api-access-jnwsh\" (UniqueName: \"kubernetes.io/projected/f66406e6-2198-443d-8130-31b590e4f566-kube-api-access-jnwsh\") pod \"f66406e6-2198-443d-8130-31b590e4f566\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") "
Oct 30 01:09:32.643904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.643816    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f66406e6-2198-443d-8130-31b590e4f566-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f66406e6-2198-443d-8130-31b590e4f566-istiod-ca-cert\") pod \"f66406e6-2198-443d-8130-31b590e4f566\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") "
Oct 30 01:09:32.644032 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.643977    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-istio-data\") pod \"f66406e6-2198-443d-8130-31b590e4f566\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") "
Oct 30 01:09:32.644032 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.644010    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-workload-certs\") pod \"f66406e6-2198-443d-8130-31b590e4f566\" (UID: \"f66406e6-2198-443d-8130-31b590e4f566\") "
Oct 30 01:09:32.644324 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.644295    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f66406e6-2198-443d-8130-31b590e4f566-istiod-ca-cert" pod "f66406e6-2198-443d-8130-31b590e4f566" (UID: "f66406e6-2198-443d-8130-31b590e4f566"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:32.644748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.644539    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-workload-socket" pod "f66406e6-2198-443d-8130-31b590e4f566" (UID: "f66406e6-2198-443d-8130-31b590e4f566"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:32.644748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.644580    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-istio-data" pod "f66406e6-2198-443d-8130-31b590e4f566" (UID: "f66406e6-2198-443d-8130-31b590e4f566"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:32.644748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.644717    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-workload-certs" pod "f66406e6-2198-443d-8130-31b590e4f566" (UID: "f66406e6-2198-443d-8130-31b590e4f566"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:32.645146 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.645123    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f66406e6-2198-443d-8130-31b590e4f566-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:32.645232 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.645152    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:32.645232 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.645170    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:32.645232 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.645183    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:32.645232 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.645196    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:32.646769 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.646742    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66406e6-2198-443d-8130-31b590e4f566-istio-token" pod "f66406e6-2198-443d-8130-31b590e4f566" (UID: "f66406e6-2198-443d-8130-31b590e4f566"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:32.647134 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.647103    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-istio-envoy" pod "f66406e6-2198-443d-8130-31b590e4f566" (UID: "f66406e6-2198-443d-8130-31b590e4f566"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:32.648552 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.648528    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f66406e6-2198-443d-8130-31b590e4f566-istio-podinfo" pod "f66406e6-2198-443d-8130-31b590e4f566" (UID: "f66406e6-2198-443d-8130-31b590e4f566"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:09:32.648774 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.648750    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66406e6-2198-443d-8130-31b590e4f566-kube-api-access-jnwsh" pod "f66406e6-2198-443d-8130-31b590e4f566" (UID: "f66406e6-2198-443d-8130-31b590e4f566"). InnerVolumeSpecName "kube-api-access-jnwsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:32.746694 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.746625    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f66406e6-2198-443d-8130-31b590e4f566-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:32.746694 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.746649    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f66406e6-2198-443d-8130-31b590e4f566-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:32.746694 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.746658    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f66406e6-2198-443d-8130-31b590e4f566-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:32.746694 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:32.746667    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnwsh\" (UniqueName: \"kubernetes.io/projected/f66406e6-2198-443d-8130-31b590e4f566-kube-api-access-jnwsh\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:33.129329 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:33.129295    6099 generic.go:356] "Generic (PLEG): container finished" podID="f66406e6-2198-443d-8130-31b590e4f566" containerID="c81ee7acbcbadc3ac360ad5f91973475a9313d8d30ba27bc037910e69b8a692c" exitCode=0
Oct 30 01:09:33.129522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:33.129336    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"
Oct 30 01:09:33.129522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:33.129383    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj" event={"ID":"f66406e6-2198-443d-8130-31b590e4f566","Type":"ContainerDied","Data":"c81ee7acbcbadc3ac360ad5f91973475a9313d8d30ba27bc037910e69b8a692c"}
Oct 30 01:09:33.129522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:33.129443    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj" event={"ID":"f66406e6-2198-443d-8130-31b590e4f566","Type":"ContainerDied","Data":"384f6b667312ee1e5225ab3fd503a2562cd30ffd9a34bf9c0f72a97d0437c0f2"}
Oct 30 01:09:33.129522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:33.129473    6099 scope.go:122] "RemoveContainer" containerID="c81ee7acbcbadc3ac360ad5f91973475a9313d8d30ba27bc037910e69b8a692c"
Oct 30 01:09:33.142315 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:33.142298    6099 scope.go:122] "RemoveContainer" containerID="c81ee7acbcbadc3ac360ad5f91973475a9313d8d30ba27bc037910e69b8a692c"
Oct 30 01:09:33.142544 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:09:33.142526    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c81ee7acbcbadc3ac360ad5f91973475a9313d8d30ba27bc037910e69b8a692c\": container with ID starting with c81ee7acbcbadc3ac360ad5f91973475a9313d8d30ba27bc037910e69b8a692c not found: ID does not exist" containerID="c81ee7acbcbadc3ac360ad5f91973475a9313d8d30ba27bc037910e69b8a692c"
Oct 30 01:09:33.142606 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:33.142557    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c81ee7acbcbadc3ac360ad5f91973475a9313d8d30ba27bc037910e69b8a692c"} err="failed to get container status \"c81ee7acbcbadc3ac360ad5f91973475a9313d8d30ba27bc037910e69b8a692c\": rpc error: code = NotFound desc = could not find container \"c81ee7acbcbadc3ac360ad5f91973475a9313d8d30ba27bc037910e69b8a692c\": container with ID starting with c81ee7acbcbadc3ac360ad5f91973475a9313d8d30ba27bc037910e69b8a692c not found: ID does not exist"
Oct 30 01:09:33.193571 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:33.193543    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"]
Oct 30 01:09:33.205155 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:33.205128    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-49dfaf-openshift-default-7967d4cccf-lcrmj"]
Oct 30 01:09:34.265370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:34.265332    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f66406e6-2198-443d-8130-31b590e4f566" path="/var/lib/kubelet/pods/f66406e6-2198-443d-8130-31b590e4f566/volumes"
Oct 30 01:09:34.637561 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:34.637518    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x"]
Oct 30 01:09:34.646740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:34.646699    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-780077-cgn7x"]
Oct 30 01:09:36.264913 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:36.264879    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bad36020-3ebe-43b5-b83e-e996bc66e201" path="/var/lib/kubelet/pods/bad36020-3ebe-43b5-b83e-e996bc66e201/volumes"
Oct 30 01:09:38.281793 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:38.281751    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj"]
Oct 30 01:09:41.585727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:41.570466    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"]
Oct 30 01:09:41.585727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:41.571195    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm" podUID="58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" containerName="istio-proxy" containerID="cri-o://90347dd1d6ea0ddd5d0d14c4faf3c82eceff97b3c18fe4dc47c2435813245d50" gracePeriod=30
Oct 30 01:09:42.058823 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.058783    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z"]
Oct 30 01:09:42.068840 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.068802    6099 cpu_manager.go:414] "containerMap: removing container" podUID="928d6088-b7d7-4052-89eb-9abba7b186a0" containerName="revoke-keys"
Oct 30 01:09:42.069098 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.069076    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="928d6088-b7d7-4052-89eb-9abba7b186a0" containerName="revoke-keys"
Oct 30 01:09:42.078844 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.078811    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e0638e9c-11e9-4aa5-b98d-15b8760b6399" containerName="revoke-keys"
Oct 30 01:09:42.079035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.079019    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0638e9c-11e9-4aa5-b98d-15b8760b6399" containerName="revoke-keys"
Oct 30 01:09:42.079169 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.079157    6099 cpu_manager.go:414] "containerMap: removing container" podUID="2e720881-a65b-4e3b-95c1-e88b04a6c88a" containerName="payload-pre-processing"
Oct 30 01:09:42.079265 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.079252    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e720881-a65b-4e3b-95c1-e88b04a6c88a" containerName="payload-pre-processing"
Oct 30 01:09:42.085727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.079367    6099 cpu_manager.go:414] "containerMap: removing container" podUID="65fe9203-ea37-4a79-9990-5ca46ceaccd0" containerName="payload-processing"
Oct 30 01:09:42.085924 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.085901    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65fe9203-ea37-4a79-9990-5ca46ceaccd0" containerName="payload-processing"
Oct 30 01:09:42.108783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.089775    6099 cpu_manager.go:414] "containerMap: removing container" podUID="f66406e6-2198-443d-8130-31b590e4f566" containerName="istio-proxy"
Oct 30 01:09:42.109072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.109043    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66406e6-2198-443d-8130-31b590e4f566" containerName="istio-proxy"
Oct 30 01:09:42.109876 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.109504    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e720881-a65b-4e3b-95c1-e88b04a6c88a" containerName="payload-pre-processing"
Oct 30 01:09:42.110062 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.110037    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="65fe9203-ea37-4a79-9990-5ca46ceaccd0" containerName="payload-processing"
Oct 30 01:09:42.110223 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.110180    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f66406e6-2198-443d-8130-31b590e4f566" containerName="istio-proxy"
Oct 30 01:09:42.110475 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.110400    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="928d6088-b7d7-4052-89eb-9abba7b186a0" containerName="revoke-keys"
Oct 30 01:09:42.110646 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.110615    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0638e9c-11e9-4aa5-b98d-15b8760b6399" containerName="revoke-keys"
Oct 30 01:09:42.142846 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.142810    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z"
Oct 30 01:09:42.160970 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.160920    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z\" (UID: \"a896ee66-7eea-4b43-9ee0-4a1c88aefd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z"
Oct 30 01:09:42.161216 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.161078    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29zp7\" (UniqueName: \"kubernetes.io/projected/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-kube-api-access-29zp7\") pod \"maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z\" (UID: \"a896ee66-7eea-4b43-9ee0-4a1c88aefd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z"
Oct 30 01:09:42.161313 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.161298    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z\" (UID: \"a896ee66-7eea-4b43-9ee0-4a1c88aefd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z"
Oct 30 01:09:42.203851 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.181085    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:09:42.203851 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.181846    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:09:42.221389 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.221343    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z"]
Oct 30 01:09:42.262756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.262331    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z\" (UID: \"a896ee66-7eea-4b43-9ee0-4a1c88aefd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z"
Oct 30 01:09:42.262756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.262454    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z\" (UID: \"a896ee66-7eea-4b43-9ee0-4a1c88aefd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z"
Oct 30 01:09:42.262756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.262580    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29zp7\" (UniqueName: \"kubernetes.io/projected/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-kube-api-access-29zp7\") pod \"maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z\" (UID: \"a896ee66-7eea-4b43-9ee0-4a1c88aefd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z"
Oct 30 01:09:42.263442 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.263313    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z\" (UID: \"a896ee66-7eea-4b43-9ee0-4a1c88aefd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z"
Oct 30 01:09:42.298533 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.298495    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z\" (UID: \"a896ee66-7eea-4b43-9ee0-4a1c88aefd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z"
Oct 30 01:09:42.300232 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.300194    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29zp7\" (UniqueName: \"kubernetes.io/projected/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-kube-api-access-29zp7\") pod \"maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z\" (UID: \"a896ee66-7eea-4b43-9ee0-4a1c88aefd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z"
Oct 30 01:09:42.329849 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.329787    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk"]
Oct 30 01:09:42.335940 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.335915    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk"
Oct 30 01:09:42.353136 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.353107    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk"]
Oct 30 01:09:42.387848 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.364592    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk\" (UID: \"2d20a4ab-aad5-4370-9bf1-8f49d75bb17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk"
Oct 30 01:09:42.387848 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.364643    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dgfx\" (UniqueName: \"kubernetes.io/projected/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-kube-api-access-5dgfx\") pod \"maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk\" (UID: \"2d20a4ab-aad5-4370-9bf1-8f49d75bb17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk"
Oct 30 01:09:42.387848 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.364780    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk\" (UID: \"2d20a4ab-aad5-4370-9bf1-8f49d75bb17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk"
Oct 30 01:09:42.492157 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.490993    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk\" (UID: \"2d20a4ab-aad5-4370-9bf1-8f49d75bb17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk"
Oct 30 01:09:42.492157 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.491036    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dgfx\" (UniqueName: \"kubernetes.io/projected/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-kube-api-access-5dgfx\") pod \"maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk\" (UID: \"2d20a4ab-aad5-4370-9bf1-8f49d75bb17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk"
Oct 30 01:09:42.492157 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.491123    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk\" (UID: \"2d20a4ab-aad5-4370-9bf1-8f49d75bb17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk"
Oct 30 01:09:42.492157 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.492040    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk\" (UID: \"2d20a4ab-aad5-4370-9bf1-8f49d75bb17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk"
Oct 30 01:09:42.495097 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.495066    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk\" (UID: \"2d20a4ab-aad5-4370-9bf1-8f49d75bb17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk"
Oct 30 01:09:42.524349 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.524324    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dgfx\" (UniqueName: \"kubernetes.io/projected/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-kube-api-access-5dgfx\") pod \"maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk\" (UID: \"2d20a4ab-aad5-4370-9bf1-8f49d75bb17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk"
Oct 30 01:09:42.543486 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.543463    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z"
Oct 30 01:09:42.656264 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.652034    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk"
Oct 30 01:09:42.913463 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:42.913435    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z"]
Oct 30 01:09:43.046567 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:43.046535    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk"]
Oct 30 01:09:43.091022 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:09:43.090983    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d20a4ab_aad5_4370_9bf1_8f49d75bb17c.slice/crio-f270860f1bf2b4089e88fff4cb5d87ed13f4afdb9cc6e54b2d10858f08bbd501 WatchSource:0}: Error finding container f270860f1bf2b4089e88fff4cb5d87ed13f4afdb9cc6e54b2d10858f08bbd501: Status 404 returned error can't find the container with id f270860f1bf2b4089e88fff4cb5d87ed13f4afdb9cc6e54b2d10858f08bbd501
Oct 30 01:09:43.286776 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:43.285867    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj" podUID="497a9701-e362-48d7-a2cd-cfa2a5a1e535" containerName="payload-processing" containerID="cri-o://d1962a41ec1d97640e65171378146e0849ce9d3de0a7674bcd4c08044fce7624" gracePeriod=25
Oct 30 01:09:43.331145 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:43.331110    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk" event={"ID":"2d20a4ab-aad5-4370-9bf1-8f49d75bb17c","Type":"ContainerStarted","Data":"f270860f1bf2b4089e88fff4cb5d87ed13f4afdb9cc6e54b2d10858f08bbd501"}
Oct 30 01:09:43.332399 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:43.332374    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z" event={"ID":"a896ee66-7eea-4b43-9ee0-4a1c88aefd89","Type":"ContainerStarted","Data":"2b20e261b9cb01249ecbbf4143986e5afbbb18c3061aeb7a8a922a729254335a"}
Oct 30 01:09:43.650809 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:43.650775    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj"
Oct 30 01:09:43.732535 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:43.732498    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497a9701-e362-48d7-a2cd-cfa2a5a1e535-kube-api-access-55btb\" (UniqueName: \"kubernetes.io/projected/497a9701-e362-48d7-a2cd-cfa2a5a1e535-kube-api-access-55btb\") pod \"497a9701-e362-48d7-a2cd-cfa2a5a1e535\" (UID: \"497a9701-e362-48d7-a2cd-cfa2a5a1e535\") "
Oct 30 01:09:43.733059 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:43.732901    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/497a9701-e362-48d7-a2cd-cfa2a5a1e535-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497a9701-e362-48d7-a2cd-cfa2a5a1e535-plugins-config-volume\") pod \"497a9701-e362-48d7-a2cd-cfa2a5a1e535\" (UID: \"497a9701-e362-48d7-a2cd-cfa2a5a1e535\") "
Oct 30 01:09:43.733344 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:43.733318    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497a9701-e362-48d7-a2cd-cfa2a5a1e535-plugins-config-volume" pod "497a9701-e362-48d7-a2cd-cfa2a5a1e535" (UID: "497a9701-e362-48d7-a2cd-cfa2a5a1e535"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:43.733748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:43.733690    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497a9701-e362-48d7-a2cd-cfa2a5a1e535-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:43.734832 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:43.734806    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497a9701-e362-48d7-a2cd-cfa2a5a1e535-kube-api-access-55btb" pod "497a9701-e362-48d7-a2cd-cfa2a5a1e535" (UID: "497a9701-e362-48d7-a2cd-cfa2a5a1e535"). InnerVolumeSpecName "kube-api-access-55btb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:43.834962 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:43.834924    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55btb\" (UniqueName: \"kubernetes.io/projected/497a9701-e362-48d7-a2cd-cfa2a5a1e535-kube-api-access-55btb\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:44.334559 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:44.334523    6099 patch_prober.go:29] interesting pod/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:09:44.334760 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:44.334584    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm" podUID="58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:09:44.341795 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:44.341721    6099 generic.go:356] "Generic (PLEG): container finished" podID="2d20a4ab-aad5-4370-9bf1-8f49d75bb17c" containerID="2e0674e7476f236d0753682c83e4fa81eb9e3378c6699a4ddcd14651935a1a9e" exitCode=0
Oct 30 01:09:44.341912 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:44.341811    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk" event={"ID":"2d20a4ab-aad5-4370-9bf1-8f49d75bb17c","Type":"ContainerDied","Data":"2e0674e7476f236d0753682c83e4fa81eb9e3378c6699a4ddcd14651935a1a9e"}
Oct 30 01:09:44.343823 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:44.343798    6099 generic.go:356] "Generic (PLEG): container finished" podID="a896ee66-7eea-4b43-9ee0-4a1c88aefd89" containerID="2f789618238245e770da18e731d025a3ecfed063efe5bebe117dbfb9bda72284" exitCode=0
Oct 30 01:09:44.343952 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:44.343881    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z" event={"ID":"a896ee66-7eea-4b43-9ee0-4a1c88aefd89","Type":"ContainerDied","Data":"2f789618238245e770da18e731d025a3ecfed063efe5bebe117dbfb9bda72284"}
Oct 30 01:09:44.345401 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:44.345381    6099 generic.go:356] "Generic (PLEG): container finished" podID="497a9701-e362-48d7-a2cd-cfa2a5a1e535" containerID="d1962a41ec1d97640e65171378146e0849ce9d3de0a7674bcd4c08044fce7624" exitCode=0
Oct 30 01:09:44.345506 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:44.345419    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj"
Oct 30 01:09:44.345506 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:44.345428    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj" event={"ID":"497a9701-e362-48d7-a2cd-cfa2a5a1e535","Type":"ContainerDied","Data":"d1962a41ec1d97640e65171378146e0849ce9d3de0a7674bcd4c08044fce7624"}
Oct 30 01:09:44.345506 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:44.345474    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj" event={"ID":"497a9701-e362-48d7-a2cd-cfa2a5a1e535","Type":"ContainerDied","Data":"f7e3e24fdd1948c747f4c68986107ea976efd1cc3e6df084ac2594d4ad8dfe22"}
Oct 30 01:09:44.345506 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:44.345497    6099 scope.go:122] "RemoveContainer" containerID="d1962a41ec1d97640e65171378146e0849ce9d3de0a7674bcd4c08044fce7624"
Oct 30 01:09:44.358791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:44.358773    6099 scope.go:122] "RemoveContainer" containerID="d1962a41ec1d97640e65171378146e0849ce9d3de0a7674bcd4c08044fce7624"
Oct 30 01:09:44.359081 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:09:44.359057    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1962a41ec1d97640e65171378146e0849ce9d3de0a7674bcd4c08044fce7624\": container with ID starting with d1962a41ec1d97640e65171378146e0849ce9d3de0a7674bcd4c08044fce7624 not found: ID does not exist" containerID="d1962a41ec1d97640e65171378146e0849ce9d3de0a7674bcd4c08044fce7624"
Oct 30 01:09:44.359199 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:44.359085    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1962a41ec1d97640e65171378146e0849ce9d3de0a7674bcd4c08044fce7624"} err="failed to get container status \"d1962a41ec1d97640e65171378146e0849ce9d3de0a7674bcd4c08044fce7624\": rpc error: code = NotFound desc = could not find container \"d1962a41ec1d97640e65171378146e0849ce9d3de0a7674bcd4c08044fce7624\": container with ID starting with d1962a41ec1d97640e65171378146e0849ce9d3de0a7674bcd4c08044fce7624 not found: ID does not exist"
Oct 30 01:09:44.458158 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:44.458115    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj"]
Oct 30 01:09:44.465622 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:44.465589    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-97ceef-684cfb864-gxprj"]
Oct 30 01:09:45.584261 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.583992    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z"
Oct 30 01:09:45.672320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.672285    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk"
Oct 30 01:09:45.678914 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.678877    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-kube-api-access-29zp7\" (UniqueName: \"kubernetes.io/projected/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-kube-api-access-29zp7\") pod \"a896ee66-7eea-4b43-9ee0-4a1c88aefd89\" (UID: \"a896ee66-7eea-4b43-9ee0-4a1c88aefd89\") "
Oct 30 01:09:45.679091 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.679069    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-maas-api-xks-ca\") pod \"a896ee66-7eea-4b43-9ee0-4a1c88aefd89\" (UID: \"a896ee66-7eea-4b43-9ee0-4a1c88aefd89\") "
Oct 30 01:09:45.679175 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.679154    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-maas-api-openshift-ca\") pod \"a896ee66-7eea-4b43-9ee0-4a1c88aefd89\" (UID: \"a896ee66-7eea-4b43-9ee0-4a1c88aefd89\") "
Oct 30 01:09:45.685065 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.685032    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-maas-api-openshift-ca" pod "a896ee66-7eea-4b43-9ee0-4a1c88aefd89" (UID: "a896ee66-7eea-4b43-9ee0-4a1c88aefd89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:45.685516 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.685487    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-kube-api-access-29zp7" pod "a896ee66-7eea-4b43-9ee0-4a1c88aefd89" (UID: "a896ee66-7eea-4b43-9ee0-4a1c88aefd89"). InnerVolumeSpecName "kube-api-access-29zp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:45.686949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.686921    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-maas-api-xks-ca" pod "a896ee66-7eea-4b43-9ee0-4a1c88aefd89" (UID: "a896ee66-7eea-4b43-9ee0-4a1c88aefd89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:45.780724 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.780678    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-maas-api-xks-ca\") pod \"2d20a4ab-aad5-4370-9bf1-8f49d75bb17c\" (UID: \"2d20a4ab-aad5-4370-9bf1-8f49d75bb17c\") "
Oct 30 01:09:45.780902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.780852    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-maas-api-openshift-ca\") pod \"2d20a4ab-aad5-4370-9bf1-8f49d75bb17c\" (UID: \"2d20a4ab-aad5-4370-9bf1-8f49d75bb17c\") "
Oct 30 01:09:45.780993 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.780959    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-kube-api-access-5dgfx\" (UniqueName: \"kubernetes.io/projected/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-kube-api-access-5dgfx\") pod \"2d20a4ab-aad5-4370-9bf1-8f49d75bb17c\" (UID: \"2d20a4ab-aad5-4370-9bf1-8f49d75bb17c\") "
Oct 30 01:09:45.781327 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.781300    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-maas-api-openshift-ca" pod "2d20a4ab-aad5-4370-9bf1-8f49d75bb17c" (UID: "2d20a4ab-aad5-4370-9bf1-8f49d75bb17c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:45.781838 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.781819    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:45.781949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.781842    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:45.781949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.781853    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29zp7\" (UniqueName: \"kubernetes.io/projected/a896ee66-7eea-4b43-9ee0-4a1c88aefd89-kube-api-access-29zp7\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:45.781949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.781863    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:45.783008 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.782984    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-maas-api-xks-ca" pod "2d20a4ab-aad5-4370-9bf1-8f49d75bb17c" (UID: "2d20a4ab-aad5-4370-9bf1-8f49d75bb17c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:45.783131 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.783105    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-kube-api-access-5dgfx" pod "2d20a4ab-aad5-4370-9bf1-8f49d75bb17c" (UID: "2d20a4ab-aad5-4370-9bf1-8f49d75bb17c"). InnerVolumeSpecName "kube-api-access-5dgfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:45.883119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.883039    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:45.883119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:45.883071    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dgfx\" (UniqueName: \"kubernetes.io/projected/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c-kube-api-access-5dgfx\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:46.263268 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:46.263196    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="497a9701-e362-48d7-a2cd-cfa2a5a1e535" path="/var/lib/kubelet/pods/497a9701-e362-48d7-a2cd-cfa2a5a1e535/volumes"
Oct 30 01:09:46.370695 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:46.370666    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z"
Oct 30 01:09:46.370900 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:46.370693    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z" event={"ID":"a896ee66-7eea-4b43-9ee0-4a1c88aefd89","Type":"ContainerDied","Data":"2b20e261b9cb01249ecbbf4143986e5afbbb18c3061aeb7a8a922a729254335a"}
Oct 30 01:09:46.370900 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:46.370768    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b20e261b9cb01249ecbbf4143986e5afbbb18c3061aeb7a8a922a729254335a"
Oct 30 01:09:46.372604 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:46.372568    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk" event={"ID":"2d20a4ab-aad5-4370-9bf1-8f49d75bb17c","Type":"ContainerDied","Data":"f270860f1bf2b4089e88fff4cb5d87ed13f4afdb9cc6e54b2d10858f08bbd501"}
Oct 30 01:09:46.372767 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:46.372620    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f270860f1bf2b4089e88fff4cb5d87ed13f4afdb9cc6e54b2d10858f08bbd501"
Oct 30 01:09:46.372767 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:46.372676    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk"
Oct 30 01:09:46.631364 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:09:46.631326    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:09:47.388857 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.388826    6099 generic.go:356] "Generic (PLEG): container finished" podID="58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" containerID="90347dd1d6ea0ddd5d0d14c4faf3c82eceff97b3c18fe4dc47c2435813245d50" exitCode=0
Oct 30 01:09:47.389049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.388921    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm" event={"ID":"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd","Type":"ContainerDied","Data":"90347dd1d6ea0ddd5d0d14c4faf3c82eceff97b3c18fe4dc47c2435813245d50"}
Oct 30 01:09:47.389049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.388988    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm" event={"ID":"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd","Type":"ContainerDied","Data":"ccb1d5005ffc34e1571c2848fe0eb3f0708ed3fe1d50a4b75349373a40ba0faf"}
Oct 30 01:09:47.389049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.389016    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccb1d5005ffc34e1571c2848fe0eb3f0708ed3fe1d50a4b75349373a40ba0faf"
Oct 30 01:09:47.389049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.389038    6099 scope.go:122] "RemoveContainer" containerID="8999a5abb33e86d850a8a12f4f903b3d49b2ee8ed22cdb3834419a048758b9fd"
Oct 30 01:09:47.427492 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.427464    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:09:47.508977 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.508698    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-data\") pod \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") "
Oct 30 01:09:47.508977 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.508797    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-token\" (UniqueName: \"kubernetes.io/projected/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-token\") pod \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") "
Oct 30 01:09:47.508977 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.508866    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-kube-api-access-vbhhh\" (UniqueName: \"kubernetes.io/projected/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-kube-api-access-vbhhh\") pod \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") "
Oct 30 01:09:47.508977 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.508906    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istiod-ca-cert\") pod \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") "
Oct 30 01:09:47.508977 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.508965    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-workload-socket\") pod \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") "
Oct 30 01:09:47.509426 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.509079    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-workload-certs\") pod \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") "
Oct 30 01:09:47.509426 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.509253    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-envoy\") pod \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") "
Oct 30 01:09:47.509426 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.509303    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-podinfo\") pod \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") "
Oct 30 01:09:47.509426 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.509333    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-credential-socket\") pod \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\" (UID: \"58ec4bd2-80f2-4c38-a59a-c36ee7b990dd\") "
Oct 30 01:09:47.509845 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.509685    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-workload-socket" pod "58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" (UID: "58ec4bd2-80f2-4c38-a59a-c36ee7b990dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:47.509845 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.509759    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-data" pod "58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" (UID: "58ec4bd2-80f2-4c38-a59a-c36ee7b990dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:47.510059 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.509959    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-credential-socket" pod "58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" (UID: "58ec4bd2-80f2-4c38-a59a-c36ee7b990dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:47.510537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.510131    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-workload-certs" pod "58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" (UID: "58ec4bd2-80f2-4c38-a59a-c36ee7b990dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:47.510537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.510455    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:47.510537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.510482    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:47.510537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.510499    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:47.510537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.510513    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:47.511019 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.510688    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istiod-ca-cert" pod "58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" (UID: "58ec4bd2-80f2-4c38-a59a-c36ee7b990dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:47.514770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.514681    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-kube-api-access-vbhhh" pod "58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" (UID: "58ec4bd2-80f2-4c38-a59a-c36ee7b990dd"). InnerVolumeSpecName "kube-api-access-vbhhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:47.514770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.514736    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-podinfo" pod "58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" (UID: "58ec4bd2-80f2-4c38-a59a-c36ee7b990dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:09:47.533920 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.530320    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-token" pod "58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" (UID: "58ec4bd2-80f2-4c38-a59a-c36ee7b990dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:47.534248 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.534216    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-envoy" pod "58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" (UID: "58ec4bd2-80f2-4c38-a59a-c36ee7b990dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:47.611580 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.611536    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:47.611580 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.611576    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:47.611580 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.611585    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:47.611845 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.611595    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbhhh\" (UniqueName: \"kubernetes.io/projected/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-kube-api-access-vbhhh\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:47.611845 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:47.611604    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:09:48.225949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:48.224761    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:09:48.225949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:48.224821    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2" podUID="caf7a449-3657-4b2e-aaa6-2cca317f241f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:09:48.399113 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:48.399029    6099 generic.go:356] "Generic (PLEG): container finished" podID="caf7a449-3657-4b2e-aaa6-2cca317f241f" containerID="c35efa95ab8313096506a40443269c2490bad57fae4fa5d348ce8f09e728f8b9" exitCode=0
Oct 30 01:09:48.399296 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:48.399083    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2" event={"ID":"caf7a449-3657-4b2e-aaa6-2cca317f241f","Type":"ContainerDied","Data":"c35efa95ab8313096506a40443269c2490bad57fae4fa5d348ce8f09e728f8b9"}
Oct 30 01:09:48.399746 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:48.399697    6099 scope.go:122] "RemoveContainer" containerID="c35efa95ab8313096506a40443269c2490bad57fae4fa5d348ce8f09e728f8b9"
Oct 30 01:09:48.401121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:48.401099    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"
Oct 30 01:09:48.401724 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:48.401681    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:09:48.401802 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:48.401784    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:09:48.401865 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:48.401835    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:09:48.528653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:48.527774    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"]
Oct 30 01:09:48.531209 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:48.531179    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-780077-openshift-default-7896f6cfc-vsvvm"]
Oct 30 01:09:49.413877 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:49.413841    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2" event={"ID":"caf7a449-3657-4b2e-aaa6-2cca317f241f","Type":"ContainerStarted","Data":"eff1c434d7c4aea6ad5c329f28a96010e88a2c97e1c894d64f4e818db5eee2de"}
Oct 30 01:09:50.263776 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:50.263746    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" path="/var/lib/kubelet/pods/58ec4bd2-80f2-4c38-a59a-c36ee7b990dd/volumes"
Oct 30 01:09:50.383058 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:50.383025    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:09:50.383425 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:50.383317    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 01:09:50.383612 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:50.383589    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2" podUID="caf7a449-3657-4b2e-aaa6-2cca317f241f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 01:09:51.383731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.382075    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 01:09:51.383731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.382148    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2" podUID="caf7a449-3657-4b2e-aaa6-2cca317f241f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 01:09:51.458685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.455384    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"]
Oct 30 01:09:51.458685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.456180    6099 cpu_manager.go:414] "containerMap: removing container" podUID="2d20a4ab-aad5-4370-9bf1-8f49d75bb17c" containerName="revoke-keys"
Oct 30 01:09:51.458685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.456196    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d20a4ab-aad5-4370-9bf1-8f49d75bb17c" containerName="revoke-keys"
Oct 30 01:09:51.458685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.456219    6099 cpu_manager.go:414] "containerMap: removing container" podUID="a896ee66-7eea-4b43-9ee0-4a1c88aefd89" containerName="revoke-keys"
Oct 30 01:09:51.458685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.456227    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a896ee66-7eea-4b43-9ee0-4a1c88aefd89" containerName="revoke-keys"
Oct 30 01:09:51.458685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.456252    6099 cpu_manager.go:414] "containerMap: removing container" podUID="58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" containerName="istio-proxy"
Oct 30 01:09:51.458685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.456257    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" containerName="istio-proxy"
Oct 30 01:09:51.458685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.456265    6099 cpu_manager.go:414] "containerMap: removing container" podUID="497a9701-e362-48d7-a2cd-cfa2a5a1e535" containerName="payload-processing"
Oct 30 01:09:51.458685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.456270    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497a9701-e362-48d7-a2cd-cfa2a5a1e535" containerName="payload-processing"
Oct 30 01:09:51.458685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.456284    6099 cpu_manager.go:414] "containerMap: removing container" podUID="58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" containerName="istio-proxy"
Oct 30 01:09:51.458685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.456291    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" containerName="istio-proxy"
Oct 30 01:09:51.458685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.456430    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d20a4ab-aad5-4370-9bf1-8f49d75bb17c" containerName="revoke-keys"
Oct 30 01:09:51.458685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.456441    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a896ee66-7eea-4b43-9ee0-4a1c88aefd89" containerName="revoke-keys"
Oct 30 01:09:51.458685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.456451    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" containerName="istio-proxy"
Oct 30 01:09:51.458685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.456459    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="497a9701-e362-48d7-a2cd-cfa2a5a1e535" containerName="payload-processing"
Oct 30 01:09:51.458685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.456470    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ec4bd2-80f2-4c38-a59a-c36ee7b990dd" containerName="istio-proxy"
Oct 30 01:09:51.499545 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.498677    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.522740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.518685    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"]
Oct 30 01:09:51.540805 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.540781    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-33e759-openshift-default-dockercfg-6b9m2\""
Oct 30 01:09:51.631510 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.630758    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/673e2d41-cb67-4956-bb96-959aa37fb924-istio-token\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.631510 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.630852    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-workload-socket\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.631510 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.630892    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/673e2d41-cb67-4956-bb96-959aa37fb924-istio-podinfo\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.631510 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.630980    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-credential-socket\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.631510 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.631030    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/673e2d41-cb67-4956-bb96-959aa37fb924-istiod-ca-cert\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.631510 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.631134    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-istio-envoy\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.631510 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.631209    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-istio-data\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.631510 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.631368    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-workload-certs\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.631510 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.631395    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stldw\" (UniqueName: \"kubernetes.io/projected/673e2d41-cb67-4956-bb96-959aa37fb924-kube-api-access-stldw\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.736782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.734997    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-workload-certs\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.736782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.735053    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stldw\" (UniqueName: \"kubernetes.io/projected/673e2d41-cb67-4956-bb96-959aa37fb924-kube-api-access-stldw\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.736782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.735091    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/673e2d41-cb67-4956-bb96-959aa37fb924-istio-token\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.736782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.735165    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-workload-socket\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.736782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.735202    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/673e2d41-cb67-4956-bb96-959aa37fb924-istio-podinfo\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.736782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.735299    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-credential-socket\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.736782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.735357    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/673e2d41-cb67-4956-bb96-959aa37fb924-istiod-ca-cert\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.736782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.735476    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-istio-envoy\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.736782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.735564    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-istio-data\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.736782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.736206    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-istio-data\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.736782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.736465    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-workload-certs\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.737555 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.737264    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-workload-socket\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.752637 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.752356    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/673e2d41-cb67-4956-bb96-959aa37fb924-istio-podinfo\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.752842 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.752671    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-credential-socket\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.759487 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.753325    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/673e2d41-cb67-4956-bb96-959aa37fb924-istiod-ca-cert\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.768788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.756720    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-istio-envoy\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.783448 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.783415    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/673e2d41-cb67-4956-bb96-959aa37fb924-istio-token\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.798738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.796403    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stldw\" (UniqueName: \"kubernetes.io/projected/673e2d41-cb67-4956-bb96-959aa37fb924-kube-api-access-stldw\") pod \"e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") " pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:51.819795 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:51.817857    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:52.195061 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:09:52.195033    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod673e2d41_cb67_4956_bb96_959aa37fb924.slice/crio-43bff4f061f176cbc1742ef7168afafb11cd8f8958168ad84e4b93a194ac1e09 WatchSource:0}: Error finding container 43bff4f061f176cbc1742ef7168afafb11cd8f8958168ad84e4b93a194ac1e09: Status 404 returned error can't find the container with id 43bff4f061f176cbc1742ef7168afafb11cd8f8958168ad84e4b93a194ac1e09
Oct 30 01:09:52.195454 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:52.195427    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"]
Oct 30 01:09:52.198128 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:52.198097    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:09:52.198211 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:52.198159    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:09:52.198211 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:52.198192    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:09:52.382728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:52.380064    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 01:09:52.382728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:52.380125    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2" podUID="caf7a449-3657-4b2e-aaa6-2cca317f241f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 01:09:52.440430 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:52.439895    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb" event={"ID":"673e2d41-cb67-4956-bb96-959aa37fb924","Type":"ContainerStarted","Data":"95bd9bbccc2e6311c11db1dc7dcb68c104620dfa0d7824adb9c15f44f4c78976"}
Oct 30 01:09:52.440430 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:52.439949    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb" event={"ID":"673e2d41-cb67-4956-bb96-959aa37fb924","Type":"ContainerStarted","Data":"43bff4f061f176cbc1742ef7168afafb11cd8f8958168ad84e4b93a194ac1e09"}
Oct 30 01:09:52.494109 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:52.487125    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb" podStartSLOduration=1.487104834 podStartE2EDuration="1.487104834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:52.487027006 +0000 UTC m=+2086.343934762" watchObservedRunningTime="2026-10-30 01:09:52.487104834 +0000 UTC m=+2086.344012590"
Oct 30 01:09:52.536302 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:52.536269    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf"]
Oct 30 01:09:52.818648 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:52.818615    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:53.388730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:53.375089    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 01:09:53.388730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:53.375148    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2" podUID="caf7a449-3657-4b2e-aaa6-2cca317f241f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 01:09:53.824434 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:53.824408    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:54.380736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:54.377014    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 01:09:54.380736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:54.377072    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2" podUID="caf7a449-3657-4b2e-aaa6-2cca317f241f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 01:09:54.459698 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:54.459669    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:54.460471 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:54.460447    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:09:55.379193 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:55.379163    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:09:55.379660 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:55.379641    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:09:55.380293 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:55.380272    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:09:56.949042 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:56.949000    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v"]
Oct 30 01:09:56.953222 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:56.953198    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v"
Oct 30 01:09:56.969971 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:56.969946    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-33e759\""
Oct 30 01:09:56.982019 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:56.981996    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-33e759-dockercfg-p5qxt\""
Oct 30 01:09:56.986585 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:56.986560    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v"]
Oct 30 01:09:57.150735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.136809    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4"]
Oct 30 01:09:57.212780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.204307    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605336de-7c5e-42ed-bc77-2f6d56507518-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v\" (UID: \"605336de-7c5e-42ed-bc77-2f6d56507518\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v"
Oct 30 01:09:57.212780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.204570    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbztr\" (UniqueName: \"kubernetes.io/projected/605336de-7c5e-42ed-bc77-2f6d56507518-kube-api-access-wbztr\") pod \"payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v\" (UID: \"605336de-7c5e-42ed-bc77-2f6d56507518\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v"
Oct 30 01:09:57.243959 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.243930    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4"]
Oct 30 01:09:57.245829 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.244253    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4"
Oct 30 01:09:57.306418 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.306387    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tj22\" (UniqueName: \"kubernetes.io/projected/28560cb7-7b8c-46ba-bf1a-afef3f320722-kube-api-access-9tj22\") pod \"payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4\" (UID: \"28560cb7-7b8c-46ba-bf1a-afef3f320722\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4"
Oct 30 01:09:57.306584 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.306538    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbztr\" (UniqueName: \"kubernetes.io/projected/605336de-7c5e-42ed-bc77-2f6d56507518-kube-api-access-wbztr\") pod \"payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v\" (UID: \"605336de-7c5e-42ed-bc77-2f6d56507518\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v"
Oct 30 01:09:57.306630 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.306580    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28560cb7-7b8c-46ba-bf1a-afef3f320722-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4\" (UID: \"28560cb7-7b8c-46ba-bf1a-afef3f320722\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4"
Oct 30 01:09:57.306799 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.306762    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605336de-7c5e-42ed-bc77-2f6d56507518-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v\" (UID: \"605336de-7c5e-42ed-bc77-2f6d56507518\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v"
Oct 30 01:09:57.307379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.307360    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605336de-7c5e-42ed-bc77-2f6d56507518-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v\" (UID: \"605336de-7c5e-42ed-bc77-2f6d56507518\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v"
Oct 30 01:09:57.335088 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.335059    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbztr\" (UniqueName: \"kubernetes.io/projected/605336de-7c5e-42ed-bc77-2f6d56507518-kube-api-access-wbztr\") pod \"payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v\" (UID: \"605336de-7c5e-42ed-bc77-2f6d56507518\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v"
Oct 30 01:09:57.408367 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.408345    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tj22\" (UniqueName: \"kubernetes.io/projected/28560cb7-7b8c-46ba-bf1a-afef3f320722-kube-api-access-9tj22\") pod \"payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4\" (UID: \"28560cb7-7b8c-46ba-bf1a-afef3f320722\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4"
Oct 30 01:09:57.408511 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.408443    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28560cb7-7b8c-46ba-bf1a-afef3f320722-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4\" (UID: \"28560cb7-7b8c-46ba-bf1a-afef3f320722\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4"
Oct 30 01:09:57.409122 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.409105    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28560cb7-7b8c-46ba-bf1a-afef3f320722-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4\" (UID: \"28560cb7-7b8c-46ba-bf1a-afef3f320722\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4"
Oct 30 01:09:57.432090 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.432065    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tj22\" (UniqueName: \"kubernetes.io/projected/28560cb7-7b8c-46ba-bf1a-afef3f320722-kube-api-access-9tj22\") pod \"payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4\" (UID: \"28560cb7-7b8c-46ba-bf1a-afef3f320722\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4"
Oct 30 01:09:57.536637 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.536552    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf" podUID="f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2" containerName="payload-pre-processing" containerID="cri-o://8d4006030d0fe5314aa32312d39dc15a8fb2a66c67de51fb813e56cf66944ce3" gracePeriod=25
Oct 30 01:09:57.546429 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.546397    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf" podUID="f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:09:57.567498 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.567470    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v"
Oct 30 01:09:57.572462 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.572445    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4"
Oct 30 01:09:57.881921 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:09:57.881891    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605336de_7c5e_42ed_bc77_2f6d56507518.slice/crio-f9a77b61742094c60eec978b3a912b2114e09f2ccb64de1b0d9c86d2796fab04 WatchSource:0}: Error finding container f9a77b61742094c60eec978b3a912b2114e09f2ccb64de1b0d9c86d2796fab04: Status 404 returned error can't find the container with id f9a77b61742094c60eec978b3a912b2114e09f2ccb64de1b0d9c86d2796fab04
Oct 30 01:09:57.882638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.882615    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v"]
Oct 30 01:09:57.894062 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:57.894038    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4"]
Oct 30 01:09:57.896191 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:09:57.896165    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28560cb7_7b8c_46ba_bf1a_afef3f320722.slice/crio-dfb6a3669ee08710da99c53de97803d1bf723e8b3938820ef9db22feb67bebce WatchSource:0}: Error finding container dfb6a3669ee08710da99c53de97803d1bf723e8b3938820ef9db22feb67bebce: Status 404 returned error can't find the container with id dfb6a3669ee08710da99c53de97803d1bf723e8b3938820ef9db22feb67bebce
Oct 30 01:09:58.498180 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:58.498143    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v" event={"ID":"605336de-7c5e-42ed-bc77-2f6d56507518","Type":"ContainerStarted","Data":"3859af8c184f6528f99344ad9bbcb868d5def482907498a3c1cdae5531b478e0"}
Oct 30 01:09:58.498733 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:58.498215    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v"
Oct 30 01:09:58.498733 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:58.498238    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v" event={"ID":"605336de-7c5e-42ed-bc77-2f6d56507518","Type":"ContainerStarted","Data":"f9a77b61742094c60eec978b3a912b2114e09f2ccb64de1b0d9c86d2796fab04"}
Oct 30 01:09:58.500029 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:58.499998    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4" event={"ID":"28560cb7-7b8c-46ba-bf1a-afef3f320722","Type":"ContainerStarted","Data":"e445628a552d272537657c320d0d158d638c40ca3a4008074e03a038bd939e4d"}
Oct 30 01:09:58.500142 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:58.500049    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4" event={"ID":"28560cb7-7b8c-46ba-bf1a-afef3f320722","Type":"ContainerStarted","Data":"dfb6a3669ee08710da99c53de97803d1bf723e8b3938820ef9db22feb67bebce"}
Oct 30 01:09:58.500142 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:58.500074    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4"
Oct 30 01:09:58.537950 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:58.537902    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v" podStartSLOduration=2.537889225 podStartE2EDuration="2.537889225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:58.536765341 +0000 UTC m=+2092.393673098" watchObservedRunningTime="2026-10-30 01:09:58.537889225 +0000 UTC m=+2092.394796979"
Oct 30 01:09:58.578831 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:58.578786    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4" podStartSLOduration=1.578774481 podStartE2EDuration="1.578774481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:58.575542049 +0000 UTC m=+2092.432449911" watchObservedRunningTime="2026-10-30 01:09:58.578774481 +0000 UTC m=+2092.435682277"
Oct 30 01:09:59.679601 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.679568    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"]
Oct 30 01:09:59.684556 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.684528    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:09:59.722115 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.706213    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"]
Oct 30 01:09:59.722115 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.710696    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-969b92\"/\"default-dockercfg-5vshj\""
Oct 30 01:09:59.722115 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.711617    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-969b92\"/\"rate-test-model-969b92-kserve-self-signed-certs\""
Oct 30 01:09:59.722115 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.718370    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-969b92\"/\"kube-root-ca.crt\""
Oct 30 01:09:59.722115 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.718451    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-969b92\"/\"openshift-service-ca.crt\""
Oct 30 01:09:59.745345 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.745314    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8af4b8ef-f359-4e4b-8474-dda967065618-tls-certs\") pod \"rate-test-model-969b92-kserve-5764b78489-r4vhc\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") " pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:09:59.745522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.745364    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-tmp-dir\") pod \"rate-test-model-969b92-kserve-5764b78489-r4vhc\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") " pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:09:59.745597 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.745520    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-dshm\") pod \"rate-test-model-969b92-kserve-5764b78489-r4vhc\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") " pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:09:59.745597 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.745567    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d2m6\" (UniqueName: \"kubernetes.io/projected/8af4b8ef-f359-4e4b-8474-dda967065618-kube-api-access-8d2m6\") pod \"rate-test-model-969b92-kserve-5764b78489-r4vhc\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") " pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:09:59.745824 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.745793    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-model-cache\") pod \"rate-test-model-969b92-kserve-5764b78489-r4vhc\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") " pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:09:59.745941 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.745848    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-home\") pod \"rate-test-model-969b92-kserve-5764b78489-r4vhc\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") " pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:09:59.847834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.847797    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-dshm\") pod \"rate-test-model-969b92-kserve-5764b78489-r4vhc\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") " pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:09:59.847834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.847844    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d2m6\" (UniqueName: \"kubernetes.io/projected/8af4b8ef-f359-4e4b-8474-dda967065618-kube-api-access-8d2m6\") pod \"rate-test-model-969b92-kserve-5764b78489-r4vhc\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") " pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:09:59.848082 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.847946    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-model-cache\") pod \"rate-test-model-969b92-kserve-5764b78489-r4vhc\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") " pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:09:59.848082 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.847985    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-home\") pod \"rate-test-model-969b92-kserve-5764b78489-r4vhc\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") " pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:09:59.848222 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.848204    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8af4b8ef-f359-4e4b-8474-dda967065618-tls-certs\") pod \"rate-test-model-969b92-kserve-5764b78489-r4vhc\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") " pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:09:59.848302 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.848236    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-tmp-dir\") pod \"rate-test-model-969b92-kserve-5764b78489-r4vhc\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") " pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:09:59.848365 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.848339    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-model-cache\") pod \"rate-test-model-969b92-kserve-5764b78489-r4vhc\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") " pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:09:59.848598 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.848577    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-tmp-dir\") pod \"rate-test-model-969b92-kserve-5764b78489-r4vhc\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") " pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:09:59.849010 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.848973    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-home\") pod \"rate-test-model-969b92-kserve-5764b78489-r4vhc\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") " pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:09:59.850338 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.850315    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-dshm\") pod \"rate-test-model-969b92-kserve-5764b78489-r4vhc\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") " pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:09:59.850927 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.850907    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8af4b8ef-f359-4e4b-8474-dda967065618-tls-certs\") pod \"rate-test-model-969b92-kserve-5764b78489-r4vhc\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") " pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:09:59.870661 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:09:59.870633    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d2m6\" (UniqueName: \"kubernetes.io/projected/8af4b8ef-f359-4e4b-8474-dda967065618-kube-api-access-8d2m6\") pod \"rate-test-model-969b92-kserve-5764b78489-r4vhc\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") " pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:10:00.018778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:00.017987    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:10:00.342276 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:10:00.342244    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8af4b8ef_f359_4e4b_8474_dda967065618.slice/crio-30c755e0bd4c36cf8ed620dc6be88f7148372f600f7b111e79191e2b2c1c67a6 WatchSource:0}: Error finding container 30c755e0bd4c36cf8ed620dc6be88f7148372f600f7b111e79191e2b2c1c67a6: Status 404 returned error can't find the container with id 30c755e0bd4c36cf8ed620dc6be88f7148372f600f7b111e79191e2b2c1c67a6
Oct 30 01:10:00.343946 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:00.343918    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"]
Oct 30 01:10:00.517156 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:00.517117    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc" event={"ID":"8af4b8ef-f359-4e4b-8474-dda967065618","Type":"ContainerStarted","Data":"96406d2129e7b1b4997c885c91499dcf119b7e2e0313d89a15b2b4f4a0c5f813"}
Oct 30 01:10:00.517330 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:00.517168    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc" event={"ID":"8af4b8ef-f359-4e4b-8474-dda967065618","Type":"ContainerStarted","Data":"30c755e0bd4c36cf8ed620dc6be88f7148372f600f7b111e79191e2b2c1c67a6"}
Oct 30 01:10:00.560630 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:00.560583    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc" podStartSLOduration=1.560569342 podStartE2EDuration="1.560569342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:00.557660985 +0000 UTC m=+2094.414568742" watchObservedRunningTime="2026-10-30 01:10:00.560569342 +0000 UTC m=+2094.417477113"
Oct 30 01:10:01.019149 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.019120    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:10:01.378649 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.378613    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"]
Oct 30 01:10:01.383160 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.383142    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.414224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.414193    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"]
Oct 30 01:10:01.487522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.487489    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x68v7\" (UniqueName: \"kubernetes.io/projected/1317f970-0225-46f8-bf6f-a5e6be570cab-kube-api-access-x68v7\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.487743 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.487564    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-credential-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.487743 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.487663    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-token\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.487889 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.487810    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1317f970-0225-46f8-bf6f-a5e6be570cab-istiod-ca-cert\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.487954 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.487884    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-data\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.488020 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.487951    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-envoy\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.488076 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.488035    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-workload-certs\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.488076 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.488069    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-podinfo\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.488173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.488115    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-workload-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.589349 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.589315    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-credential-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.589503 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.589358    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-token\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.589503 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.589413    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1317f970-0225-46f8-bf6f-a5e6be570cab-istiod-ca-cert\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.589503 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.589438    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-data\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.589503 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.589491    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-envoy\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.589747 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.589530    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-workload-certs\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.589747 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.589562    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-podinfo\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.589747 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.589605    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-workload-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.589929 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.589771    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x68v7\" (UniqueName: \"kubernetes.io/projected/1317f970-0225-46f8-bf6f-a5e6be570cab-kube-api-access-x68v7\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.589929 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.589772    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-credential-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.590048 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.589969    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-workload-certs\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.590048 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.589990    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-data\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.590175 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.590156    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1317f970-0225-46f8-bf6f-a5e6be570cab-istiod-ca-cert\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.590216 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.590180    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-workload-socket\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.592063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.592042    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-envoy\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.592266 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.592244    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-podinfo\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.612318 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.612297    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-token\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.612732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.612694    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x68v7\" (UniqueName: \"kubernetes.io/projected/1317f970-0225-46f8-bf6f-a5e6be570cab-kube-api-access-x68v7\") pod \"e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") " pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:01.698899 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:01.698829    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:02.016969 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:10:02.016939    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1317f970_0225_46f8_bf6f_a5e6be570cab.slice/crio-c24c867875a769bc7d2b0c798a42bed7f06b90dddf695565a8aa500bc98f4e72 WatchSource:0}: Error finding container c24c867875a769bc7d2b0c798a42bed7f06b90dddf695565a8aa500bc98f4e72: Status 404 returned error can't find the container with id c24c867875a769bc7d2b0c798a42bed7f06b90dddf695565a8aa500bc98f4e72
Oct 30 01:10:02.018127 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:02.018099    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"]
Oct 30 01:10:02.019984 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:02.019949    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:10:02.020268 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:02.020017    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:10:02.020268 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:02.020046    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:10:02.537385 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:02.537349    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q" event={"ID":"1317f970-0225-46f8-bf6f-a5e6be570cab","Type":"ContainerStarted","Data":"20caef8c899bfcfb47124e154244780b0d70ac48ed00fa805eaaa18e547d8646"}
Oct 30 01:10:02.537613 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:02.537412    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q" event={"ID":"1317f970-0225-46f8-bf6f-a5e6be570cab","Type":"ContainerStarted","Data":"c24c867875a769bc7d2b0c798a42bed7f06b90dddf695565a8aa500bc98f4e72"}
Oct 30 01:10:02.591775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:02.591699    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q" podStartSLOduration=1.59168467 podStartE2EDuration="1.59168467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:02.591212471 +0000 UTC m=+2096.448120226" watchObservedRunningTime="2026-10-30 01:10:02.59168467 +0000 UTC m=+2096.448592425"
Oct 30 01:10:02.699298 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:02.699266    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:03.701888 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:03.701851    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Oct 30 01:10:03.702295 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:03.701909    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q" podUID="1317f970-0225-46f8-bf6f-a5e6be570cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Oct 30 01:10:04.707730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:04.701840    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Oct 30 01:10:04.707730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:04.701899    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q" podUID="1317f970-0225-46f8-bf6f-a5e6be570cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Oct 30 01:10:05.144447 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:05.144416    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr"]
Oct 30 01:10:05.156356 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:05.156331    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-97ceef-4wqfr"]
Oct 30 01:10:05.700230 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:05.700194    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Oct 30 01:10:05.700417 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:05.700251    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q" podUID="1317f970-0225-46f8-bf6f-a5e6be570cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Oct 30 01:10:06.264530 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:06.264496    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="928d6088-b7d7-4052-89eb-9abba7b186a0" path="/var/lib/kubelet/pods/928d6088-b7d7-4052-89eb-9abba7b186a0/volumes"
Oct 30 01:10:06.596867 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:06.596834    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"]
Oct 30 01:10:06.597114 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:06.597066    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n" podUID="571da8ac-1d91-472b-80a1-187c16a687e8" containerName="maas-api" containerID="cri-o://05ab48848676da1718c31a1e7d209cb25f8bcdcf8e6c1854e66fea6565d444ae" gracePeriod=30
Oct 30 01:10:06.699176 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:06.699140    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Oct 30 01:10:06.699360 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:06.699195    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q" podUID="1317f970-0225-46f8-bf6f-a5e6be570cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Oct 30 01:10:06.727264 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:06.727232    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 01:10:06.735282 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:06.735259    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 01:10:07.419015 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.418985    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"
Oct 30 01:10:07.450036 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.450003    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk"]
Oct 30 01:10:07.453928 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.453901    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf3466-9sgpk"]
Oct 30 01:10:07.498821 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.498685    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/571da8ac-1d91-472b-80a1-187c16a687e8-kube-api-access-kw4tm\" (UniqueName: \"kubernetes.io/projected/571da8ac-1d91-472b-80a1-187c16a687e8-kube-api-access-kw4tm\") pod \"571da8ac-1d91-472b-80a1-187c16a687e8\" (UID: \"571da8ac-1d91-472b-80a1-187c16a687e8\") "
Oct 30 01:10:07.498996 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.498879    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/571da8ac-1d91-472b-80a1-187c16a687e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/571da8ac-1d91-472b-80a1-187c16a687e8-metrics-certs\") pod \"571da8ac-1d91-472b-80a1-187c16a687e8\" (UID: \"571da8ac-1d91-472b-80a1-187c16a687e8\") "
Oct 30 01:10:07.498996 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.498910    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/571da8ac-1d91-472b-80a1-187c16a687e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/571da8ac-1d91-472b-80a1-187c16a687e8-openshift-service-ca\") pod \"571da8ac-1d91-472b-80a1-187c16a687e8\" (UID: \"571da8ac-1d91-472b-80a1-187c16a687e8\") "
Oct 30 01:10:07.499171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.499154    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/571da8ac-1d91-472b-80a1-187c16a687e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/571da8ac-1d91-472b-80a1-187c16a687e8-maas-api-tls\") pod \"571da8ac-1d91-472b-80a1-187c16a687e8\" (UID: \"571da8ac-1d91-472b-80a1-187c16a687e8\") "
Oct 30 01:10:07.502383 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.502350    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/571da8ac-1d91-472b-80a1-187c16a687e8-maas-api-tls" pod "571da8ac-1d91-472b-80a1-187c16a687e8" (UID: "571da8ac-1d91-472b-80a1-187c16a687e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:10:07.502587 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.502496    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/571da8ac-1d91-472b-80a1-187c16a687e8-metrics-certs" pod "571da8ac-1d91-472b-80a1-187c16a687e8" (UID: "571da8ac-1d91-472b-80a1-187c16a687e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:10:07.502765 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.502698    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/571da8ac-1d91-472b-80a1-187c16a687e8-openshift-service-ca" pod "571da8ac-1d91-472b-80a1-187c16a687e8" (UID: "571da8ac-1d91-472b-80a1-187c16a687e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:07.503363 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.503339    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571da8ac-1d91-472b-80a1-187c16a687e8-kube-api-access-kw4tm" pod "571da8ac-1d91-472b-80a1-187c16a687e8" (UID: "571da8ac-1d91-472b-80a1-187c16a687e8"). InnerVolumeSpecName "kube-api-access-kw4tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:07.546760 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.546700    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf" podUID="f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:10:07.601526 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.601449    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/571da8ac-1d91-472b-80a1-187c16a687e8-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:07.601830 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.601808    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/571da8ac-1d91-472b-80a1-187c16a687e8-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:07.601968 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.601943    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/571da8ac-1d91-472b-80a1-187c16a687e8-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:07.602079 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.602066    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw4tm\" (UniqueName: \"kubernetes.io/projected/571da8ac-1d91-472b-80a1-187c16a687e8-kube-api-access-kw4tm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:07.633596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.633443    6099 generic.go:356] "Generic (PLEG): container finished" podID="571da8ac-1d91-472b-80a1-187c16a687e8" containerID="05ab48848676da1718c31a1e7d209cb25f8bcdcf8e6c1854e66fea6565d444ae" exitCode=0
Oct 30 01:10:07.633596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.633592    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n" event={"ID":"571da8ac-1d91-472b-80a1-187c16a687e8","Type":"ContainerDied","Data":"05ab48848676da1718c31a1e7d209cb25f8bcdcf8e6c1854e66fea6565d444ae"}
Oct 30 01:10:07.633869 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.633634    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n" event={"ID":"571da8ac-1d91-472b-80a1-187c16a687e8","Type":"ContainerDied","Data":"bdf62cfd52e22e01dceaedb2dbaffe79b367a3a1e977c9b40bfef182cf748111"}
Oct 30 01:10:07.633869 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.633656    6099 scope.go:122] "RemoveContainer" containerID="05ab48848676da1718c31a1e7d209cb25f8bcdcf8e6c1854e66fea6565d444ae"
Oct 30 01:10:07.633869 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.633835    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"
Oct 30 01:10:07.653095 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.653076    6099 scope.go:122] "RemoveContainer" containerID="05ab48848676da1718c31a1e7d209cb25f8bcdcf8e6c1854e66fea6565d444ae"
Oct 30 01:10:07.653372 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:10:07.653354    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ab48848676da1718c31a1e7d209cb25f8bcdcf8e6c1854e66fea6565d444ae\": container with ID starting with 05ab48848676da1718c31a1e7d209cb25f8bcdcf8e6c1854e66fea6565d444ae not found: ID does not exist" containerID="05ab48848676da1718c31a1e7d209cb25f8bcdcf8e6c1854e66fea6565d444ae"
Oct 30 01:10:07.653484 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.653381    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ab48848676da1718c31a1e7d209cb25f8bcdcf8e6c1854e66fea6565d444ae"} err="failed to get container status \"05ab48848676da1718c31a1e7d209cb25f8bcdcf8e6c1854e66fea6565d444ae\": rpc error: code = NotFound desc = could not find container \"05ab48848676da1718c31a1e7d209cb25f8bcdcf8e6c1854e66fea6565d444ae\": container with ID starting with 05ab48848676da1718c31a1e7d209cb25f8bcdcf8e6c1854e66fea6565d444ae not found: ID does not exist"
Oct 30 01:10:07.696192 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.696159    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"]
Oct 30 01:10:07.699992 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.699965    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cf3466-5946b54fc5-j4g9n"]
Oct 30 01:10:07.700104 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.699991    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Oct 30 01:10:07.700104 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:07.700032    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q" podUID="1317f970-0225-46f8-bf6f-a5e6be570cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Oct 30 01:10:08.263577 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:08.263544    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d20a4ab-aad5-4370-9bf1-8f49d75bb17c" path="/var/lib/kubelet/pods/2d20a4ab-aad5-4370-9bf1-8f49d75bb17c/volumes"
Oct 30 01:10:08.264088 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:08.264072    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="571da8ac-1d91-472b-80a1-187c16a687e8" path="/var/lib/kubelet/pods/571da8ac-1d91-472b-80a1-187c16a687e8/volumes"
Oct 30 01:10:08.886956 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:08.886923    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"]
Oct 30 01:10:08.887841 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:08.887814    6099 cpu_manager.go:414] "containerMap: removing container" podUID="571da8ac-1d91-472b-80a1-187c16a687e8" containerName="maas-api"
Oct 30 01:10:08.887841 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:08.887834    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="571da8ac-1d91-472b-80a1-187c16a687e8" containerName="maas-api"
Oct 30 01:10:08.892035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:08.892006    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="571da8ac-1d91-472b-80a1-187c16a687e8" containerName="maas-api"
Oct 30 01:10:08.904993 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:08.904966    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:08.907582 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:08.907556    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:08.907976 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:08.907949    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:08.909308 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:08.909284    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:10:08.915801 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:08.915309    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"]
Oct 30 01:10:08.923203 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:08.923174    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"]
Oct 30 01:10:08.923398 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:08.923285    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:08.935099 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:08.935077    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"]
Oct 30 01:10:08.990522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:08.990496    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-16c1bb-openshift-default-dockercfg-h52zt\""
Oct 30 01:10:09.023431 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.023395    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.023666 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.023442    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.023666 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.023470    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxh96\" (UniqueName: \"kubernetes.io/projected/227245db-8ac3-47c1-ae8c-ba5ff96202c6-kube-api-access-kxh96\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.023666 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.023620    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.023885 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.023676    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26062a6-cf7a-4faf-a955-5e408bc47c08-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.023885 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.023764    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.023885 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.023875    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.024012 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.023989    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.024067 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.024039    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bh8n\" (UniqueName: \"kubernetes.io/projected/f26062a6-cf7a-4faf-a955-5e408bc47c08-kube-api-access-9bh8n\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.024194 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.024179    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.024262 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.024231    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.024318 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.024296    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.024356 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.024343    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.024473 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.024455    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.024516 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.024489    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.024628 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.024615    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.024665 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.024640    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.024665 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.024659    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.126430 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.126397    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.126430 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.126432    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.126689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.126452    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.126689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.126470    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.126689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.126489    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.126689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.126525    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.126689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.126547    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.126689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.126565    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxh96\" (UniqueName: \"kubernetes.io/projected/227245db-8ac3-47c1-ae8c-ba5ff96202c6-kube-api-access-kxh96\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.126689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.126620    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.126689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.126680    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26062a6-cf7a-4faf-a955-5e408bc47c08-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.127152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.126759    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.127152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.126858    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.127152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.126882    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.127152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.126939    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.127152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.126882    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.127152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.126995    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.127152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.127002    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.127152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.127010    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.127152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.127037    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bh8n\" (UniqueName: \"kubernetes.io/projected/f26062a6-cf7a-4faf-a955-5e408bc47c08-kube-api-access-9bh8n\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.127152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.127147    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.127723 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.127193    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.127723 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.127200    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.127723 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.127243    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.127723 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.127293    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.127723 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.127418    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.127723 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.127639    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26062a6-cf7a-4faf-a955-5e408bc47c08-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.128090 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.128050    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.128156 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.128127    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.129280 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.129253    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.129527 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.129509    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.129603 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.129582    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.129694 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.129680    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.149843 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.149787    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.150163 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.150147    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.151211 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.151184    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxh96\" (UniqueName: \"kubernetes.io/projected/227245db-8ac3-47c1-ae8c-ba5ff96202c6-kube-api-access-kxh96\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.151324 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.151303    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bh8n\" (UniqueName: \"kubernetes.io/projected/f26062a6-cf7a-4faf-a955-5e408bc47c08-kube-api-access-9bh8n\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.258573 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.258542    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:09.261717 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.261682    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:09.510222 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.510148    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v"
Oct 30 01:10:09.510398 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.510224    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4"
Oct 30 01:10:09.598152 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:10:09.598122    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26062a6_cf7a_4faf_a955_5e408bc47c08.slice/crio-207d9c1179a96db2e71751c9de5c5755775905c610606fee9eee435d03b269d9 WatchSource:0}: Error finding container 207d9c1179a96db2e71751c9de5c5755775905c610606fee9eee435d03b269d9: Status 404 returned error can't find the container with id 207d9c1179a96db2e71751c9de5c5755775905c610606fee9eee435d03b269d9
Oct 30 01:10:09.598985 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.598959    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"]
Oct 30 01:10:09.603249 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.603207    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:10:09.603384 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.603295    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:10:09.603384 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.603342    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:10:09.609776 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:10:09.609556    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227245db_8ac3_47c1_ae8c_ba5ff96202c6.slice/crio-a5b266f020b1b7679d2ffc43bd61b1b72b757e4c8b84210eb45c495f7c288e64 WatchSource:0}: Error finding container a5b266f020b1b7679d2ffc43bd61b1b72b757e4c8b84210eb45c495f7c288e64: Status 404 returned error can't find the container with id a5b266f020b1b7679d2ffc43bd61b1b72b757e4c8b84210eb45c495f7c288e64
Oct 30 01:10:09.611860 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.611837    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"]
Oct 30 01:10:09.616970 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.616937    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:10:09.617095 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.617027    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:10:09.617095 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.617059    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:10:09.659456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.659427    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq" event={"ID":"f26062a6-cf7a-4faf-a955-5e408bc47c08","Type":"ContainerStarted","Data":"207d9c1179a96db2e71751c9de5c5755775905c610606fee9eee435d03b269d9"}
Oct 30 01:10:09.661051 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:09.661028    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb" event={"ID":"227245db-8ac3-47c1-ae8c-ba5ff96202c6","Type":"ContainerStarted","Data":"a5b266f020b1b7679d2ffc43bd61b1b72b757e4c8b84210eb45c495f7c288e64"}
Oct 30 01:10:10.019410 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:10.019371    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:10:10.032578 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:10.032545    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:10:10.713732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:10.712720    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq" event={"ID":"f26062a6-cf7a-4faf-a955-5e408bc47c08","Type":"ContainerStarted","Data":"93d5c66001633e84d2952abc81e3193a7f97d5f060bee02e676d8b62ecc0481c"}
Oct 30 01:10:10.718201 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:10.718165    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb" event={"ID":"227245db-8ac3-47c1-ae8c-ba5ff96202c6","Type":"ContainerStarted","Data":"06f34b48462002ac3e7cb3748338d0bd297ec9a09a1ac0eba95d515a2510f330"}
Oct 30 01:10:10.731694 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:10.731670    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:10:10.821745 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:10.821349    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq" podStartSLOduration=2.821327479 podStartE2EDuration="2.821327479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:10.787996997 +0000 UTC m=+2104.644904754" watchObservedRunningTime="2026-10-30 01:10:10.821327479 +0000 UTC m=+2104.678235237"
Oct 30 01:10:10.852964 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:10.852861    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb" podStartSLOduration=2.852847452 podStartE2EDuration="2.852847452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:10.851471516 +0000 UTC m=+2104.708379271" watchObservedRunningTime="2026-10-30 01:10:10.852847452 +0000 UTC m=+2104.709755207"
Oct 30 01:10:11.258944 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:11.258869    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:11.260504 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:11.260471    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 01:10:11.260657 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:11.260518    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb" podUID="227245db-8ac3-47c1-ae8c-ba5ff96202c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 01:10:11.262749 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:11.262734    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:11.263898 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:11.263873    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Oct 30 01:10:11.264021 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:11.263917    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq" podUID="f26062a6-cf7a-4faf-a955-5e408bc47c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Oct 30 01:10:12.259155 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:12.259125    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 01:10:12.260085 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:12.259698    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb" podUID="227245db-8ac3-47c1-ae8c-ba5ff96202c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 01:10:12.262438 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:12.262408    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Oct 30 01:10:12.262567 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:12.262456    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq" podUID="f26062a6-cf7a-4faf-a955-5e408bc47c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Oct 30 01:10:12.342493 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:12.341581    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"]
Oct 30 01:10:12.342493 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:12.341835    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5" podUID="cb207b92-e133-428b-99b2-e17d2e903511" containerName="istio-proxy" containerID="cri-o://ca9799fa667b3812e1d8a87d11a2f76fc5059a155ccdf9d0082145959358afaf" gracePeriod=30
Oct 30 01:10:13.085731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:13.083242    6099 scope.go:122] "RemoveContainer" containerID="90347dd1d6ea0ddd5d0d14c4faf3c82eceff97b3c18fe4dc47c2435813245d50"
Oct 30 01:10:13.259367 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:13.259336    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 01:10:13.259764 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:13.259407    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb" podUID="227245db-8ac3-47c1-ae8c-ba5ff96202c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 01:10:13.262692 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:13.262669    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Oct 30 01:10:13.262791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:13.262748    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq" podUID="f26062a6-cf7a-4faf-a955-5e408bc47c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Oct 30 01:10:14.258855 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:14.258825    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 01:10:14.259033 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:14.258886    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb" podUID="227245db-8ac3-47c1-ae8c-ba5ff96202c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 01:10:14.262884 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:14.262856    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Oct 30 01:10:14.263217 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:14.262892    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq" podUID="f26062a6-cf7a-4faf-a955-5e408bc47c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Oct 30 01:10:15.259323 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:15.259283    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 01:10:15.259501 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:15.259336    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb" podUID="227245db-8ac3-47c1-ae8c-ba5ff96202c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 01:10:15.262650 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:15.262557    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Oct 30 01:10:15.262798 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:15.262687    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq" podUID="f26062a6-cf7a-4faf-a955-5e408bc47c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Oct 30 01:10:15.422726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:15.366581    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"]
Oct 30 01:10:15.437548 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:15.437497    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c" podUID="502b2376-6a3e-4c6d-9cab-aa704bcd3053" containerName="istio-proxy" containerID="cri-o://6a4b44f6633b75b933f8c95ab507ac883796deab0fe5b7cc83ac1985234f43d3" gracePeriod=30
Oct 30 01:10:15.909813 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:15.909773    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc"]
Oct 30 01:10:15.927821 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:15.927792    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc"
Oct 30 01:10:15.931165 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:15.931132    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc"]
Oct 30 01:10:16.008898 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:15.992279    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:10:16.012047 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:16.009920    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc\" (UID: \"6aba30ff-23fb-4bbb-a858-562cd2a67ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc"
Oct 30 01:10:16.012445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:16.012422    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdk5g\" (UniqueName: \"kubernetes.io/projected/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-kube-api-access-jdk5g\") pod \"maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc\" (UID: \"6aba30ff-23fb-4bbb-a858-562cd2a67ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc"
Oct 30 01:10:16.012688 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:16.012665    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc\" (UID: \"6aba30ff-23fb-4bbb-a858-562cd2a67ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc"
Oct 30 01:10:16.013171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:16.010140    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:10:16.115067 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:16.115028    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc\" (UID: \"6aba30ff-23fb-4bbb-a858-562cd2a67ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc"
Oct 30 01:10:16.115255 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:16.115140    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdk5g\" (UniqueName: \"kubernetes.io/projected/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-kube-api-access-jdk5g\") pod \"maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc\" (UID: \"6aba30ff-23fb-4bbb-a858-562cd2a67ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc"
Oct 30 01:10:16.115255 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:16.115175    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc\" (UID: \"6aba30ff-23fb-4bbb-a858-562cd2a67ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc"
Oct 30 01:10:16.116089 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:16.116059    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc\" (UID: \"6aba30ff-23fb-4bbb-a858-562cd2a67ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc"
Oct 30 01:10:16.117811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:16.117773    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc\" (UID: \"6aba30ff-23fb-4bbb-a858-562cd2a67ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc"
Oct 30 01:10:16.139059 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:16.139024    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdk5g\" (UniqueName: \"kubernetes.io/projected/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-kube-api-access-jdk5g\") pod \"maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc\" (UID: \"6aba30ff-23fb-4bbb-a858-562cd2a67ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc"
Oct 30 01:10:16.247017 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:16.246936    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc"
Oct 30 01:10:16.260259 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:16.260224    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 01:10:16.260418 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:16.260291    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb" podUID="227245db-8ac3-47c1-ae8c-ba5ff96202c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 01:10:16.263231 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:16.263192    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Oct 30 01:10:16.263394 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:16.263240    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq" podUID="f26062a6-cf7a-4faf-a955-5e408bc47c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Oct 30 01:10:16.576117 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:16.575904    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc"]
Oct 30 01:10:16.647329 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:10:16.646604    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:10:16.878855 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:16.878722    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc" event={"ID":"6aba30ff-23fb-4bbb-a858-562cd2a67ac2","Type":"ContainerStarted","Data":"0555d96635ef43f9001ce51c24b20bcba64e48f455c30f8e84a499fc96bd81e8"}
Oct 30 01:10:17.191555 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.191469    6099 patch_prober.go:29] interesting pod/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:10:17.191555 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.191537    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5" podUID="cb207b92-e133-428b-99b2-e17d2e903511" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:10:17.264924 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.264893    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:17.265921 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.265356    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:17.266269 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.266251    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:10:17.268361 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.268338    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:17.268644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.268625    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:17.269367 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.269349    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:10:17.548018 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:10:17.547984    6099 logging.go:55] [core] [Channel #4769 SubChannel #4770]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.209:50052", ServerName: "10.134.0.209:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.209:50052: connect: connection refused"
Oct 30 01:10:17.704296 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.704265    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:10:17.822943 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.822840    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf"
Oct 30 01:10:17.849247 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.849218    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-istio-envoy\") pod \"cb207b92-e133-428b-99b2-e17d2e903511\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") "
Oct 30 01:10:17.849391 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.849338    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb207b92-e133-428b-99b2-e17d2e903511-istio-token\" (UniqueName: \"kubernetes.io/projected/cb207b92-e133-428b-99b2-e17d2e903511-istio-token\") pod \"cb207b92-e133-428b-99b2-e17d2e903511\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") "
Oct 30 01:10:17.849461 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.849394    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb207b92-e133-428b-99b2-e17d2e903511-kube-api-access-9xvtd\" (UniqueName: \"kubernetes.io/projected/cb207b92-e133-428b-99b2-e17d2e903511-kube-api-access-9xvtd\") pod \"cb207b92-e133-428b-99b2-e17d2e903511\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") "
Oct 30 01:10:17.849461 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.849446    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-workload-certs\") pod \"cb207b92-e133-428b-99b2-e17d2e903511\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") "
Oct 30 01:10:17.849600 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.849585    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-workload-socket\") pod \"cb207b92-e133-428b-99b2-e17d2e903511\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") "
Oct 30 01:10:17.849733 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.849700    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-credential-socket\") pod \"cb207b92-e133-428b-99b2-e17d2e903511\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") "
Oct 30 01:10:17.849803 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.849789    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb207b92-e133-428b-99b2-e17d2e903511-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb207b92-e133-428b-99b2-e17d2e903511-istiod-ca-cert\") pod \"cb207b92-e133-428b-99b2-e17d2e903511\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") "
Oct 30 01:10:17.849859 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.849793    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-workload-certs" pod "cb207b92-e133-428b-99b2-e17d2e903511" (UID: "cb207b92-e133-428b-99b2-e17d2e903511"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:17.849925 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.849865    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-istio-data\") pod \"cb207b92-e133-428b-99b2-e17d2e903511\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") "
Oct 30 01:10:17.849979 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.849938    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb207b92-e133-428b-99b2-e17d2e903511-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb207b92-e133-428b-99b2-e17d2e903511-istio-podinfo\") pod \"cb207b92-e133-428b-99b2-e17d2e903511\" (UID: \"cb207b92-e133-428b-99b2-e17d2e903511\") "
Oct 30 01:10:17.849979 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.849962    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-workload-socket" pod "cb207b92-e133-428b-99b2-e17d2e903511" (UID: "cb207b92-e133-428b-99b2-e17d2e903511"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:17.850231 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.850209    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb207b92-e133-428b-99b2-e17d2e903511-istiod-ca-cert" pod "cb207b92-e133-428b-99b2-e17d2e903511" (UID: "cb207b92-e133-428b-99b2-e17d2e903511"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:17.850558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.850339    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-credential-socket" pod "cb207b92-e133-428b-99b2-e17d2e903511" (UID: "cb207b92-e133-428b-99b2-e17d2e903511"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:17.850558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.850458    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-istio-data" pod "cb207b92-e133-428b-99b2-e17d2e903511" (UID: "cb207b92-e133-428b-99b2-e17d2e903511"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:17.850846 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.850826    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:17.850920 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.850850    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:17.850920 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.850864    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb207b92-e133-428b-99b2-e17d2e903511-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:17.850920 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.850878    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:17.850920 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.850890    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:17.852066 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.852038    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-istio-envoy" pod "cb207b92-e133-428b-99b2-e17d2e903511" (UID: "cb207b92-e133-428b-99b2-e17d2e903511"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:17.852165 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.852055    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb207b92-e133-428b-99b2-e17d2e903511-kube-api-access-9xvtd" pod "cb207b92-e133-428b-99b2-e17d2e903511" (UID: "cb207b92-e133-428b-99b2-e17d2e903511"). InnerVolumeSpecName "kube-api-access-9xvtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:17.852497 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.852472    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb207b92-e133-428b-99b2-e17d2e903511-istio-podinfo" pod "cb207b92-e133-428b-99b2-e17d2e903511" (UID: "cb207b92-e133-428b-99b2-e17d2e903511"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:10:17.853125 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.853104    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb207b92-e133-428b-99b2-e17d2e903511-istio-token" pod "cb207b92-e133-428b-99b2-e17d2e903511" (UID: "cb207b92-e133-428b-99b2-e17d2e903511"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:17.894020 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.893982    6099 generic.go:356] "Generic (PLEG): container finished" podID="6aba30ff-23fb-4bbb-a858-562cd2a67ac2" containerID="9dcdb469d40edb9167ed8f5786ad507539e9e1a2eadcf70accc6f74b1662534b" exitCode=0
Oct 30 01:10:17.894197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.894065    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc" event={"ID":"6aba30ff-23fb-4bbb-a858-562cd2a67ac2","Type":"ContainerDied","Data":"9dcdb469d40edb9167ed8f5786ad507539e9e1a2eadcf70accc6f74b1662534b"}
Oct 30 01:10:17.896441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.896415    6099 generic.go:356] "Generic (PLEG): container finished" podID="f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2" containerID="8d4006030d0fe5314aa32312d39dc15a8fb2a66c67de51fb813e56cf66944ce3" exitCode=0
Oct 30 01:10:17.896565 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.896474    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf"
Oct 30 01:10:17.896565 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.896509    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf" event={"ID":"f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2","Type":"ContainerDied","Data":"8d4006030d0fe5314aa32312d39dc15a8fb2a66c67de51fb813e56cf66944ce3"}
Oct 30 01:10:17.896663 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.896567    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf" event={"ID":"f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2","Type":"ContainerDied","Data":"0428a29125c6f8f0787e4aa17803de5c5649d3b92789ef05b886d8fd54e16bf8"}
Oct 30 01:10:17.896663 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.896599    6099 scope.go:122] "RemoveContainer" containerID="8d4006030d0fe5314aa32312d39dc15a8fb2a66c67de51fb813e56cf66944ce3"
Oct 30 01:10:17.899245 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.899208    6099 generic.go:356] "Generic (PLEG): container finished" podID="cb207b92-e133-428b-99b2-e17d2e903511" containerID="ca9799fa667b3812e1d8a87d11a2f76fc5059a155ccdf9d0082145959358afaf" exitCode=0
Oct 30 01:10:17.900330 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.900311    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"
Oct 30 01:10:17.904757 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.903815    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5" event={"ID":"cb207b92-e133-428b-99b2-e17d2e903511","Type":"ContainerDied","Data":"ca9799fa667b3812e1d8a87d11a2f76fc5059a155ccdf9d0082145959358afaf"}
Oct 30 01:10:17.904757 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.903868    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5" event={"ID":"cb207b92-e133-428b-99b2-e17d2e903511","Type":"ContainerDied","Data":"cccb2d83678a473231acc92c027201b19cebb3d57d20e5a58c3dca27e26cca2f"}
Oct 30 01:10:17.908945 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.908925    6099 scope.go:122] "RemoveContainer" containerID="8d4006030d0fe5314aa32312d39dc15a8fb2a66c67de51fb813e56cf66944ce3"
Oct 30 01:10:17.909334 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:10:17.909303    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4006030d0fe5314aa32312d39dc15a8fb2a66c67de51fb813e56cf66944ce3\": container with ID starting with 8d4006030d0fe5314aa32312d39dc15a8fb2a66c67de51fb813e56cf66944ce3 not found: ID does not exist" containerID="8d4006030d0fe5314aa32312d39dc15a8fb2a66c67de51fb813e56cf66944ce3"
Oct 30 01:10:17.909478 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.909343    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4006030d0fe5314aa32312d39dc15a8fb2a66c67de51fb813e56cf66944ce3"} err="failed to get container status \"8d4006030d0fe5314aa32312d39dc15a8fb2a66c67de51fb813e56cf66944ce3\": rpc error: code = NotFound desc = could not find container \"8d4006030d0fe5314aa32312d39dc15a8fb2a66c67de51fb813e56cf66944ce3\": container with ID starting with 8d4006030d0fe5314aa32312d39dc15a8fb2a66c67de51fb813e56cf66944ce3 not found: ID does not exist"
Oct 30 01:10:17.909478 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.909367    6099 scope.go:122] "RemoveContainer" containerID="ca9799fa667b3812e1d8a87d11a2f76fc5059a155ccdf9d0082145959358afaf"
Oct 30 01:10:17.921765 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.921740    6099 scope.go:122] "RemoveContainer" containerID="3f9aa1642154abcfdac8ab90306762bc06602008b969c02c9c6bb449a6320bb7"
Oct 30 01:10:17.932399 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.932378    6099 scope.go:122] "RemoveContainer" containerID="ca9799fa667b3812e1d8a87d11a2f76fc5059a155ccdf9d0082145959358afaf"
Oct 30 01:10:17.932701 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:10:17.932675    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9799fa667b3812e1d8a87d11a2f76fc5059a155ccdf9d0082145959358afaf\": container with ID starting with ca9799fa667b3812e1d8a87d11a2f76fc5059a155ccdf9d0082145959358afaf not found: ID does not exist" containerID="ca9799fa667b3812e1d8a87d11a2f76fc5059a155ccdf9d0082145959358afaf"
Oct 30 01:10:17.932826 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.932743    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9799fa667b3812e1d8a87d11a2f76fc5059a155ccdf9d0082145959358afaf"} err="failed to get container status \"ca9799fa667b3812e1d8a87d11a2f76fc5059a155ccdf9d0082145959358afaf\": rpc error: code = NotFound desc = could not find container \"ca9799fa667b3812e1d8a87d11a2f76fc5059a155ccdf9d0082145959358afaf\": container with ID starting with ca9799fa667b3812e1d8a87d11a2f76fc5059a155ccdf9d0082145959358afaf not found: ID does not exist"
Oct 30 01:10:17.932826 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.932770    6099 scope.go:122] "RemoveContainer" containerID="3f9aa1642154abcfdac8ab90306762bc06602008b969c02c9c6bb449a6320bb7"
Oct 30 01:10:17.933082 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:10:17.933062    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9aa1642154abcfdac8ab90306762bc06602008b969c02c9c6bb449a6320bb7\": container with ID starting with 3f9aa1642154abcfdac8ab90306762bc06602008b969c02c9c6bb449a6320bb7 not found: ID does not exist" containerID="3f9aa1642154abcfdac8ab90306762bc06602008b969c02c9c6bb449a6320bb7"
Oct 30 01:10:17.933128 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.933088    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9aa1642154abcfdac8ab90306762bc06602008b969c02c9c6bb449a6320bb7"} err="failed to get container status \"3f9aa1642154abcfdac8ab90306762bc06602008b969c02c9c6bb449a6320bb7\": rpc error: code = NotFound desc = could not find container \"3f9aa1642154abcfdac8ab90306762bc06602008b969c02c9c6bb449a6320bb7\": container with ID starting with 3f9aa1642154abcfdac8ab90306762bc06602008b969c02c9c6bb449a6320bb7 not found: ID does not exist"
Oct 30 01:10:17.952220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.952187    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2-plugins-config-volume\") pod \"f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2\" (UID: \"f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2\") "
Oct 30 01:10:17.952562 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.952538    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2-plugins-config-volume" pod "f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2" (UID: "f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:17.952613 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.952549    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2-kube-api-access-dt82p\" (UniqueName: \"kubernetes.io/projected/f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2-kube-api-access-dt82p\") pod \"f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2\" (UID: \"f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2\") "
Oct 30 01:10:17.953173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.953155    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb207b92-e133-428b-99b2-e17d2e903511-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:17.953173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.953173    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb207b92-e133-428b-99b2-e17d2e903511-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:17.953316 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.953182    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb207b92-e133-428b-99b2-e17d2e903511-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:17.953316 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.953191    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xvtd\" (UniqueName: \"kubernetes.io/projected/cb207b92-e133-428b-99b2-e17d2e903511-kube-api-access-9xvtd\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:17.953316 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.953200    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:17.955099 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.955069    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2-kube-api-access-dt82p" pod "f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2" (UID: "f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2"). InnerVolumeSpecName "kube-api-access-dt82p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:17.995259 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.995178    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"]
Oct 30 01:10:17.999397 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:17.999375    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-97ceef-openshift-default-77d68d9ff9-2rqf5"]
Oct 30 01:10:18.054666 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:18.054642    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt82p\" (UniqueName: \"kubernetes.io/projected/f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2-kube-api-access-dt82p\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:18.246970 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:18.246896    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf"]
Oct 30 01:10:18.264921 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:18.264889    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb207b92-e133-428b-99b2-e17d2e903511" path="/var/lib/kubelet/pods/cb207b92-e133-428b-99b2-e17d2e903511/volumes"
Oct 30 01:10:18.265261 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:18.265246    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf"]
Oct 30 01:10:18.545873 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:18.545836    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf3466-56cf7bd867-xltzf" podUID="f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.209:50052\" within 1s: context deadline exceeded"
Oct 30 01:10:19.105356 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.105324    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc"
Oct 30 01:10:19.177997 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.177902    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-maas-api-xks-ca\") pod \"6aba30ff-23fb-4bbb-a858-562cd2a67ac2\" (UID: \"6aba30ff-23fb-4bbb-a858-562cd2a67ac2\") "
Oct 30 01:10:19.178184 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.178144    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-kube-api-access-jdk5g\" (UniqueName: \"kubernetes.io/projected/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-kube-api-access-jdk5g\") pod \"6aba30ff-23fb-4bbb-a858-562cd2a67ac2\" (UID: \"6aba30ff-23fb-4bbb-a858-562cd2a67ac2\") "
Oct 30 01:10:19.178296 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.178262    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-maas-api-openshift-ca\") pod \"6aba30ff-23fb-4bbb-a858-562cd2a67ac2\" (UID: \"6aba30ff-23fb-4bbb-a858-562cd2a67ac2\") "
Oct 30 01:10:19.178625 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.178596    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-maas-api-openshift-ca" pod "6aba30ff-23fb-4bbb-a858-562cd2a67ac2" (UID: "6aba30ff-23fb-4bbb-a858-562cd2a67ac2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:19.179131 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.179107    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:19.180674 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.180646    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-maas-api-xks-ca" pod "6aba30ff-23fb-4bbb-a858-562cd2a67ac2" (UID: "6aba30ff-23fb-4bbb-a858-562cd2a67ac2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:10:19.181109 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.181087    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-kube-api-access-jdk5g" pod "6aba30ff-23fb-4bbb-a858-562cd2a67ac2" (UID: "6aba30ff-23fb-4bbb-a858-562cd2a67ac2"). InnerVolumeSpecName "kube-api-access-jdk5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:19.281198 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.281168    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:19.281198 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.281195    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdk5g\" (UniqueName: \"kubernetes.io/projected/6aba30ff-23fb-4bbb-a858-562cd2a67ac2-kube-api-access-jdk5g\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:19.737479 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.737445    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d"]
Oct 30 01:10:19.738172 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.738157    6099 cpu_manager.go:414] "containerMap: removing container" podUID="f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2" containerName="payload-pre-processing"
Oct 30 01:10:19.738229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.738173    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2" containerName="payload-pre-processing"
Oct 30 01:10:19.738229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.738217    6099 cpu_manager.go:414] "containerMap: removing container" podUID="6aba30ff-23fb-4bbb-a858-562cd2a67ac2" containerName="revoke-keys"
Oct 30 01:10:19.738229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.738222    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aba30ff-23fb-4bbb-a858-562cd2a67ac2" containerName="revoke-keys"
Oct 30 01:10:19.738328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.738235    6099 cpu_manager.go:414] "containerMap: removing container" podUID="cb207b92-e133-428b-99b2-e17d2e903511" containerName="istio-proxy"
Oct 30 01:10:19.738328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.738240    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb207b92-e133-428b-99b2-e17d2e903511" containerName="istio-proxy"
Oct 30 01:10:19.738328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.738248    6099 cpu_manager.go:414] "containerMap: removing container" podUID="cb207b92-e133-428b-99b2-e17d2e903511" containerName="istio-proxy"
Oct 30 01:10:19.738328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.738252    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb207b92-e133-428b-99b2-e17d2e903511" containerName="istio-proxy"
Oct 30 01:10:19.738451 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.738358    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aba30ff-23fb-4bbb-a858-562cd2a67ac2" containerName="revoke-keys"
Oct 30 01:10:19.738451 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.738368    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb207b92-e133-428b-99b2-e17d2e903511" containerName="istio-proxy"
Oct 30 01:10:19.738451 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.738374    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb207b92-e133-428b-99b2-e17d2e903511" containerName="istio-proxy"
Oct 30 01:10:19.738451 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.738383    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2" containerName="payload-pre-processing"
Oct 30 01:10:19.742325 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.742308    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d"
Oct 30 01:10:19.759234 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.759200    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-d01638\""
Oct 30 01:10:19.761208 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.761180    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d"]
Oct 30 01:10:19.768244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.768223    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-d01638-dockercfg-7zzt9\""
Oct 30 01:10:19.906239 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.906197    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzr82\" (UniqueName: \"kubernetes.io/projected/e0d9984d-a336-4b7d-a72c-11a183059b3e-kube-api-access-xzr82\") pod \"payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d\" (UID: \"e0d9984d-a336-4b7d-a72c-11a183059b3e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d"
Oct 30 01:10:19.906467 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.906328    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d9984d-a336-4b7d-a72c-11a183059b3e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d\" (UID: \"e0d9984d-a336-4b7d-a72c-11a183059b3e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d"
Oct 30 01:10:19.954729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.952607    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc" event={"ID":"6aba30ff-23fb-4bbb-a858-562cd2a67ac2","Type":"ContainerDied","Data":"0555d96635ef43f9001ce51c24b20bcba64e48f455c30f8e84a499fc96bd81e8"}
Oct 30 01:10:19.954729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.952671    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0555d96635ef43f9001ce51c24b20bcba64e48f455c30f8e84a499fc96bd81e8"
Oct 30 01:10:19.954729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:19.952768    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc"
Oct 30 01:10:20.013690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.013581    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzr82\" (UniqueName: \"kubernetes.io/projected/e0d9984d-a336-4b7d-a72c-11a183059b3e-kube-api-access-xzr82\") pod \"payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d\" (UID: \"e0d9984d-a336-4b7d-a72c-11a183059b3e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d"
Oct 30 01:10:20.014471 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.014440    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d9984d-a336-4b7d-a72c-11a183059b3e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d\" (UID: \"e0d9984d-a336-4b7d-a72c-11a183059b3e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d"
Oct 30 01:10:20.016353 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.016202    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d9984d-a336-4b7d-a72c-11a183059b3e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d\" (UID: \"e0d9984d-a336-4b7d-a72c-11a183059b3e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d"
Oct 30 01:10:20.064292 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.063423    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzr82\" (UniqueName: \"kubernetes.io/projected/e0d9984d-a336-4b7d-a72c-11a183059b3e-kube-api-access-xzr82\") pod \"payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d\" (UID: \"e0d9984d-a336-4b7d-a72c-11a183059b3e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d"
Oct 30 01:10:20.264861 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.264787    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2" path="/var/lib/kubelet/pods/f10e5a00-8ac3-4382-948f-b9fe4b3ea9c2/volumes"
Oct 30 01:10:20.357450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.357416    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d"
Oct 30 01:10:20.729284 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.729247    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d"]
Oct 30 01:10:20.742022 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:10:20.741990    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d9984d_a336_4b7d_a72c_11a183059b3e.slice/crio-f8510269b69a2c88d9fe199e8664276cc047a4f5e5bed64854abb726d2cfeb0d WatchSource:0}: Error finding container f8510269b69a2c88d9fe199e8664276cc047a4f5e5bed64854abb726d2cfeb0d: Status 404 returned error can't find the container with id f8510269b69a2c88d9fe199e8664276cc047a4f5e5bed64854abb726d2cfeb0d
Oct 30 01:10:20.805669 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.805646    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:10:20.832762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.832740    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-credential-socket\") pod \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") "
Oct 30 01:10:20.832855 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.832834    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istiod-ca-cert\") pod \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") "
Oct 30 01:10:20.832909 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.832859    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-podinfo\") pod \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") "
Oct 30 01:10:20.832909 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.832881    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-token\" (UniqueName: \"kubernetes.io/projected/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-token\") pod \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") "
Oct 30 01:10:20.832909 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.832902    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-workload-socket\") pod \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") "
Oct 30 01:10:20.833065 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.832928    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-envoy\") pod \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") "
Oct 30 01:10:20.833065 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.832965    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-credential-socket" pod "502b2376-6a3e-4c6d-9cab-aa704bcd3053" (UID: "502b2376-6a3e-4c6d-9cab-aa704bcd3053"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:20.833182 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.833062    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-workload-certs\") pod \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") "
Oct 30 01:10:20.833182 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.833121    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-data\") pod \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") "
Oct 30 01:10:20.833333 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.833312    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/502b2376-6a3e-4c6d-9cab-aa704bcd3053-kube-api-access-nwhb7\" (UniqueName: \"kubernetes.io/projected/502b2376-6a3e-4c6d-9cab-aa704bcd3053-kube-api-access-nwhb7\") pod \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\" (UID: \"502b2376-6a3e-4c6d-9cab-aa704bcd3053\") "
Oct 30 01:10:20.833333 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.833256    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istiod-ca-cert" pod "502b2376-6a3e-4c6d-9cab-aa704bcd3053" (UID: "502b2376-6a3e-4c6d-9cab-aa704bcd3053"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:20.833454 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.833340    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-workload-certs" pod "502b2376-6a3e-4c6d-9cab-aa704bcd3053" (UID: "502b2376-6a3e-4c6d-9cab-aa704bcd3053"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:20.833454 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.833419    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-workload-socket" pod "502b2376-6a3e-4c6d-9cab-aa704bcd3053" (UID: "502b2376-6a3e-4c6d-9cab-aa704bcd3053"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:20.833578 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.833523    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-data" pod "502b2376-6a3e-4c6d-9cab-aa704bcd3053" (UID: "502b2376-6a3e-4c6d-9cab-aa704bcd3053"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:20.834054 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.834033    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:20.834170 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.834061    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:20.834170 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.834075    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:20.834170 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.834089    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:20.834170 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.834102    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:20.834989 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.834969    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-podinfo" pod "502b2376-6a3e-4c6d-9cab-aa704bcd3053" (UID: "502b2376-6a3e-4c6d-9cab-aa704bcd3053"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:10:20.835247 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.835225    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-token" pod "502b2376-6a3e-4c6d-9cab-aa704bcd3053" (UID: "502b2376-6a3e-4c6d-9cab-aa704bcd3053"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:20.836106 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.836086    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-envoy" pod "502b2376-6a3e-4c6d-9cab-aa704bcd3053" (UID: "502b2376-6a3e-4c6d-9cab-aa704bcd3053"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:20.836185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.836113    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/502b2376-6a3e-4c6d-9cab-aa704bcd3053-kube-api-access-nwhb7" pod "502b2376-6a3e-4c6d-9cab-aa704bcd3053" (UID: "502b2376-6a3e-4c6d-9cab-aa704bcd3053"). InnerVolumeSpecName "kube-api-access-nwhb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:20.936006 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.935976    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwhb7\" (UniqueName: \"kubernetes.io/projected/502b2376-6a3e-4c6d-9cab-aa704bcd3053-kube-api-access-nwhb7\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:20.936006 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.936002    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:20.936006 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.936012    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:20.936234 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.936022    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/502b2376-6a3e-4c6d-9cab-aa704bcd3053-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:20.965516 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.965484    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d" event={"ID":"e0d9984d-a336-4b7d-a72c-11a183059b3e","Type":"ContainerStarted","Data":"0119e0d0d2f7926c607f7f1aa144236c90eceb006a20187e514443a8594a1fc6"}
Oct 30 01:10:20.965772 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.965542    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d" event={"ID":"e0d9984d-a336-4b7d-a72c-11a183059b3e","Type":"ContainerStarted","Data":"f8510269b69a2c88d9fe199e8664276cc047a4f5e5bed64854abb726d2cfeb0d"}
Oct 30 01:10:20.965772 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.965750    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d"
Oct 30 01:10:20.967203 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.967169    6099 generic.go:356] "Generic (PLEG): container finished" podID="502b2376-6a3e-4c6d-9cab-aa704bcd3053" containerID="6a4b44f6633b75b933f8c95ab507ac883796deab0fe5b7cc83ac1985234f43d3" exitCode=0
Oct 30 01:10:20.967351 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.967215    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"
Oct 30 01:10:20.967351 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.967247    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c" event={"ID":"502b2376-6a3e-4c6d-9cab-aa704bcd3053","Type":"ContainerDied","Data":"6a4b44f6633b75b933f8c95ab507ac883796deab0fe5b7cc83ac1985234f43d3"}
Oct 30 01:10:20.967351 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.967296    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c" event={"ID":"502b2376-6a3e-4c6d-9cab-aa704bcd3053","Type":"ContainerDied","Data":"ae6857c3d8fa9c8bc6d5e0abfc30af759248154a8ba366c9678319999128b73c"}
Oct 30 01:10:20.967351 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.967320    6099 scope.go:122] "RemoveContainer" containerID="6a4b44f6633b75b933f8c95ab507ac883796deab0fe5b7cc83ac1985234f43d3"
Oct 30 01:10:20.981146 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.981125    6099 scope.go:122] "RemoveContainer" containerID="846f119fc4eeffdcc2136c24c8139230a91c411b2c2fdad8a5891e0877a8fabb"
Oct 30 01:10:20.991314 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.991291    6099 scope.go:122] "RemoveContainer" containerID="6a4b44f6633b75b933f8c95ab507ac883796deab0fe5b7cc83ac1985234f43d3"
Oct 30 01:10:20.991593 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:10:20.991573    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a4b44f6633b75b933f8c95ab507ac883796deab0fe5b7cc83ac1985234f43d3\": container with ID starting with 6a4b44f6633b75b933f8c95ab507ac883796deab0fe5b7cc83ac1985234f43d3 not found: ID does not exist" containerID="6a4b44f6633b75b933f8c95ab507ac883796deab0fe5b7cc83ac1985234f43d3"
Oct 30 01:10:20.991655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.991601    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4b44f6633b75b933f8c95ab507ac883796deab0fe5b7cc83ac1985234f43d3"} err="failed to get container status \"6a4b44f6633b75b933f8c95ab507ac883796deab0fe5b7cc83ac1985234f43d3\": rpc error: code = NotFound desc = could not find container \"6a4b44f6633b75b933f8c95ab507ac883796deab0fe5b7cc83ac1985234f43d3\": container with ID starting with 6a4b44f6633b75b933f8c95ab507ac883796deab0fe5b7cc83ac1985234f43d3 not found: ID does not exist"
Oct 30 01:10:20.991655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.991618    6099 scope.go:122] "RemoveContainer" containerID="846f119fc4eeffdcc2136c24c8139230a91c411b2c2fdad8a5891e0877a8fabb"
Oct 30 01:10:20.991895 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:10:20.991873    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"846f119fc4eeffdcc2136c24c8139230a91c411b2c2fdad8a5891e0877a8fabb\": container with ID starting with 846f119fc4eeffdcc2136c24c8139230a91c411b2c2fdad8a5891e0877a8fabb not found: ID does not exist" containerID="846f119fc4eeffdcc2136c24c8139230a91c411b2c2fdad8a5891e0877a8fabb"
Oct 30 01:10:20.991956 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:20.991902    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846f119fc4eeffdcc2136c24c8139230a91c411b2c2fdad8a5891e0877a8fabb"} err="failed to get container status \"846f119fc4eeffdcc2136c24c8139230a91c411b2c2fdad8a5891e0877a8fabb\": rpc error: code = NotFound desc = could not find container \"846f119fc4eeffdcc2136c24c8139230a91c411b2c2fdad8a5891e0877a8fabb\": container with ID starting with 846f119fc4eeffdcc2136c24c8139230a91c411b2c2fdad8a5891e0877a8fabb not found: ID does not exist"
Oct 30 01:10:21.011278 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:21.011235    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d" podStartSLOduration=2.011221537 podStartE2EDuration="2.011221537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:21.007988836 +0000 UTC m=+2114.864896593" watchObservedRunningTime="2026-10-30 01:10:21.011221537 +0000 UTC m=+2114.868129292"
Oct 30 01:10:21.052279 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:21.052246    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"]
Oct 30 01:10:21.060255 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:21.060228    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-cf3466-openshift-default-97cd47b87-cls2c"]
Oct 30 01:10:22.268757 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:22.268724    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="502b2376-6a3e-4c6d-9cab-aa704bcd3053" path="/var/lib/kubelet/pods/502b2376-6a3e-4c6d-9cab-aa704bcd3053/volumes"
Oct 30 01:10:23.933505 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:23.933456    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"]
Oct 30 01:10:23.936464 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:23.936435    6099 cpu_manager.go:414] "containerMap: removing container" podUID="502b2376-6a3e-4c6d-9cab-aa704bcd3053" containerName="istio-proxy"
Oct 30 01:10:23.936464 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:23.936463    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="502b2376-6a3e-4c6d-9cab-aa704bcd3053" containerName="istio-proxy"
Oct 30 01:10:23.936641 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:23.936618    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="502b2376-6a3e-4c6d-9cab-aa704bcd3053" containerName="istio-proxy"
Oct 30 01:10:23.936641 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:23.936632    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="502b2376-6a3e-4c6d-9cab-aa704bcd3053" containerName="istio-proxy"
Oct 30 01:10:23.943730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:23.942246    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:23.957794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:23.957645    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"]
Oct 30 01:10:23.963432 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:23.963409    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-ae06fe\"/\"openshift-service-ca.crt\""
Oct 30 01:10:23.964362 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:23.964342    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-ae06fe\"/\"kube-root-ca.crt\""
Oct 30 01:10:23.968588 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:23.968565    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-ae06fe\"/\"default-dockercfg-tfpg9\""
Oct 30 01:10:23.968788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:23.968769    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-ae06fe\"/\"rate-test-model-ae06fe-kserve-self-signed-certs\""
Oct 30 01:10:23.997534 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:23.997499    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-model-cache\") pod \"rate-test-model-ae06fe-kserve-6c44c678df-pgswl\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:23.997922 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:23.997895    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz6t5\" (UniqueName: \"kubernetes.io/projected/657a9d22-491f-4ef1-909b-a27511860627-kube-api-access-xz6t5\") pod \"rate-test-model-ae06fe-kserve-6c44c678df-pgswl\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:23.998146 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:23.998127    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-home\") pod \"rate-test-model-ae06fe-kserve-6c44c678df-pgswl\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:23.998304 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:23.998290    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-tmp-dir\") pod \"rate-test-model-ae06fe-kserve-6c44c678df-pgswl\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:23.999741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:23.999721    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-dshm\") pod \"rate-test-model-ae06fe-kserve-6c44c678df-pgswl\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:24.000078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.000044    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657a9d22-491f-4ef1-909b-a27511860627-tls-certs\") pod \"rate-test-model-ae06fe-kserve-6c44c678df-pgswl\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:24.077676 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.066678    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"]
Oct 30 01:10:24.077676 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.067899    6099 cpu_manager.go:414] "containerMap: removing container" podUID="502b2376-6a3e-4c6d-9cab-aa704bcd3053" containerName="istio-proxy"
Oct 30 01:10:24.077676 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.067921    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="502b2376-6a3e-4c6d-9cab-aa704bcd3053" containerName="istio-proxy"
Oct 30 01:10:24.103234 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.103199    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.106992 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.105369    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-model-cache\") pod \"rate-test-model-ae06fe-kserve-6c44c678df-pgswl\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:24.106992 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.105452    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz6t5\" (UniqueName: \"kubernetes.io/projected/657a9d22-491f-4ef1-909b-a27511860627-kube-api-access-xz6t5\") pod \"rate-test-model-ae06fe-kserve-6c44c678df-pgswl\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:24.106992 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.105501    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-home\") pod \"rate-test-model-ae06fe-kserve-6c44c678df-pgswl\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:24.106992 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.105525    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-tmp-dir\") pod \"rate-test-model-ae06fe-kserve-6c44c678df-pgswl\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:24.106992 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.105649    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-dshm\") pod \"rate-test-model-ae06fe-kserve-6c44c678df-pgswl\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:24.109534 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.109504    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"]
Oct 30 01:10:24.110244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.110217    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-model-cache\") pod \"rate-test-model-ae06fe-kserve-6c44c678df-pgswl\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:24.112518 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.112492    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657a9d22-491f-4ef1-909b-a27511860627-tls-certs\") pod \"rate-test-model-ae06fe-kserve-6c44c678df-pgswl\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:24.113515 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.113486    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-tmp-dir\") pod \"rate-test-model-ae06fe-kserve-6c44c678df-pgswl\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:24.113771 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.113750    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-home\") pod \"rate-test-model-ae06fe-kserve-6c44c678df-pgswl\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:24.129741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.127843    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657a9d22-491f-4ef1-909b-a27511860627-tls-certs\") pod \"rate-test-model-ae06fe-kserve-6c44c678df-pgswl\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:24.129741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.129246    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-dshm\") pod \"rate-test-model-ae06fe-kserve-6c44c678df-pgswl\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:24.133669 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.133637    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz6t5\" (UniqueName: \"kubernetes.io/projected/657a9d22-491f-4ef1-909b-a27511860627-kube-api-access-xz6t5\") pod \"rate-test-model-ae06fe-kserve-6c44c678df-pgswl\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") " pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:24.217778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.216819    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.217778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.216922    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.217778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.216976    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.217778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.217112    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.217778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.217149    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.217778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.217317    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e0b7e93-d524-4443-98ca-a0993b9232c0-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.217778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.217346    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw8k8\" (UniqueName: \"kubernetes.io/projected/3e0b7e93-d524-4443-98ca-a0993b9232c0-kube-api-access-mw8k8\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.217778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.217406    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.217778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.217498    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.290043 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.289999    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:24.319140 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.319102    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e0b7e93-d524-4443-98ca-a0993b9232c0-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.319316 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.319153    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw8k8\" (UniqueName: \"kubernetes.io/projected/3e0b7e93-d524-4443-98ca-a0993b9232c0-kube-api-access-mw8k8\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.319316 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.319208    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.319316 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.319291    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.319507 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.319355    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.319507 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.319431    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.319507 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.319482    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.319667 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.319602    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.319667 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.319630    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.320191 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.320163    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-credential-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.320828 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.320805    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e0b7e93-d524-4443-98ca-a0993b9232c0-istiod-ca-cert\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.321572 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.321544    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-workload-certs\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.323042 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.323016    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-workload-socket\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.323678 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.323652    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-data\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.324211 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.324183    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-envoy\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.325333 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.325312    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-podinfo\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.344124 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.344092    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw8k8\" (UniqueName: \"kubernetes.io/projected/3e0b7e93-d524-4443-98ca-a0993b9232c0-kube-api-access-mw8k8\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.344369 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.344350    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-token\") pod \"e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") " pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.479040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.478951    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z"]
Oct 30 01:10:24.483040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.483013    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1da7b2-jk59z"]
Oct 30 01:10:24.593854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.593822    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:24.649103 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.647115    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"]
Oct 30 01:10:24.951032 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:10:24.950991    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0b7e93_d524_4443_98ca_a0993b9232c0.slice/crio-d89c4d02e6e7a22dab5c771add1b173ff5fac1447fa9668eaeea56e3d576b0a1 WatchSource:0}: Error finding container d89c4d02e6e7a22dab5c771add1b173ff5fac1447fa9668eaeea56e3d576b0a1: Status 404 returned error can't find the container with id d89c4d02e6e7a22dab5c771add1b173ff5fac1447fa9668eaeea56e3d576b0a1
Oct 30 01:10:24.951887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.951860    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"]
Oct 30 01:10:24.953699 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.953665    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:10:24.953846 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.953765    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:10:24.953846 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:24.953802    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:10:25.025457 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:25.025430    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62" event={"ID":"3e0b7e93-d524-4443-98ca-a0993b9232c0","Type":"ContainerStarted","Data":"d89c4d02e6e7a22dab5c771add1b173ff5fac1447fa9668eaeea56e3d576b0a1"}
Oct 30 01:10:25.027480 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:25.027456    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl" event={"ID":"657a9d22-491f-4ef1-909b-a27511860627","Type":"ContainerStarted","Data":"0512c0c70d00efd83c33301fb73d737f9d170b74629dafa00fbe9cef882c0b99"}
Oct 30 01:10:25.027601 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:25.027490    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl" event={"ID":"657a9d22-491f-4ef1-909b-a27511860627","Type":"ContainerStarted","Data":"e76c9d6cb5fa813163696aa22e5382b69d484bceedc24f00bfbb0da805ac1f44"}
Oct 30 01:10:25.069891 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:25.069840    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl" podStartSLOduration=2.069821197 podStartE2EDuration="2.069821197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:25.069243146 +0000 UTC m=+2118.926150898" watchObservedRunningTime="2026-10-30 01:10:25.069821197 +0000 UTC m=+2118.926728955"
Oct 30 01:10:25.291839 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:25.291807    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:26.038926 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:26.038883    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62" event={"ID":"3e0b7e93-d524-4443-98ca-a0993b9232c0","Type":"ContainerStarted","Data":"ad59b0471e3cce4ece2a3bc1e68f215f6d08b2bef3920668a883439a0c2a4b58"}
Oct 30 01:10:26.116473 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:26.116417    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62" podStartSLOduration=2.116399155 podStartE2EDuration="2.116399155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:26.113217704 +0000 UTC m=+2119.970125461" watchObservedRunningTime="2026-10-30 01:10:26.116399155 +0000 UTC m=+2119.973306912"
Oct 30 01:10:26.279877 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:26.265380    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a896ee66-7eea-4b43-9ee0-4a1c88aefd89" path="/var/lib/kubelet/pods/a896ee66-7eea-4b43-9ee0-4a1c88aefd89/volumes"
Oct 30 01:10:26.594311 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:26.594275    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:26.596373 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:26.596349    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body=
Oct 30 01:10:26.596537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:26.596392    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62" podUID="3e0b7e93-d524-4443-98ca-a0993b9232c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused"
Oct 30 01:10:27.594411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:27.594372    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body=
Oct 30 01:10:27.594828 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:27.594431    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62" podUID="3e0b7e93-d524-4443-98ca-a0993b9232c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused"
Oct 30 01:10:28.594459 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:28.594423    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body=
Oct 30 01:10:28.594937 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:28.594479    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62" podUID="3e0b7e93-d524-4443-98ca-a0993b9232c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused"
Oct 30 01:10:29.594345 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:29.594303    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body=
Oct 30 01:10:29.594548 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:29.594378    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62" podUID="3e0b7e93-d524-4443-98ca-a0993b9232c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused"
Oct 30 01:10:30.546741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.546694    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"]
Oct 30 01:10:30.552795 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.552774    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"
Oct 30 01:10:30.595821 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.595775    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body=
Oct 30 01:10:30.596290 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.595844    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62" podUID="3e0b7e93-d524-4443-98ca-a0993b9232c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused"
Oct 30 01:10:30.596879 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.596849    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-33e759-serving-cert\""
Oct 30 01:10:30.603119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.603091    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"]
Oct 30 01:10:30.644780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.644742    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq\" (UID: \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"
Oct 30 01:10:30.644919 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.644795    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq\" (UID: \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"
Oct 30 01:10:30.645073 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.645045    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq\" (UID: \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"
Oct 30 01:10:30.645347 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.645323    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ms2z\" (UniqueName: \"kubernetes.io/projected/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-kube-api-access-7ms2z\") pod \"maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq\" (UID: \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"
Oct 30 01:10:30.746608 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.746566    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ms2z\" (UniqueName: \"kubernetes.io/projected/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-kube-api-access-7ms2z\") pod \"maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq\" (UID: \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"
Oct 30 01:10:30.746861 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.746675    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq\" (UID: \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"
Oct 30 01:10:30.746861 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.746741    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq\" (UID: \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"
Oct 30 01:10:30.746961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.746914    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq\" (UID: \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"
Oct 30 01:10:30.747845 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.747815    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq\" (UID: \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"
Oct 30 01:10:30.749336 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.749303    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq\" (UID: \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"
Oct 30 01:10:30.749467 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.749339    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq\" (UID: \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"
Oct 30 01:10:30.794291 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.794251    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ms2z\" (UniqueName: \"kubernetes.io/projected/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-kube-api-access-7ms2z\") pod \"maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq\" (UID: \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"
Oct 30 01:10:30.888595 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:30.888510    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"
Oct 30 01:10:31.304921 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:31.304866    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"]
Oct 30 01:10:31.305915 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:10:31.305844    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8abb9d_eff6_478e_9801_c37fd4ef0b7f.slice/crio-123a31df02b5f5ed9faf3bc5c6547cbe40dfcc9c111819792830ccd4b05352fe WatchSource:0}: Error finding container 123a31df02b5f5ed9faf3bc5c6547cbe40dfcc9c111819792830ccd4b05352fe: Status 404 returned error can't find the container with id 123a31df02b5f5ed9faf3bc5c6547cbe40dfcc9c111819792830ccd4b05352fe
Oct 30 01:10:31.570986 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:31.570911    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"]
Oct 30 01:10:31.571363 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:31.571333    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8" podUID="7451c167-4bdb-44dd-96ab-ed72782229d0" containerName="istio-proxy" containerID="cri-o://c01b4f1bf335b3e4bcba294183c93c533fd0fb245fe7c0c52e3e29c843a245be" gracePeriod=30
Oct 30 01:10:31.594233 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:31.594202    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body=
Oct 30 01:10:31.594410 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:31.594251    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62" podUID="3e0b7e93-d524-4443-98ca-a0993b9232c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused"
Oct 30 01:10:31.735728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:31.728476    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:10:31.996821 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:31.992591    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d"
Oct 30 01:10:32.149362 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:32.149262    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq" event={"ID":"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f","Type":"ContainerStarted","Data":"81fbfdfda6083ffa89e4e812aa1041b549094d4c2433102be60ac5535db9dc27"}
Oct 30 01:10:32.149362 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:32.149332    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq" event={"ID":"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f","Type":"ContainerStarted","Data":"123a31df02b5f5ed9faf3bc5c6547cbe40dfcc9c111819792830ccd4b05352fe"}
Oct 30 01:10:32.149549 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:32.149379    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"
Oct 30 01:10:32.246681 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:32.244263    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq" podStartSLOduration=1.881273188 podStartE2EDuration="2.244242091s" totalImagesPullingTime="362.968903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:32.199242013 +0000 UTC m=+2126.056149792" watchObservedRunningTime="2026-10-30 01:10:32.244242091 +0000 UTC m=+2126.101149850"
Oct 30 01:10:32.602763 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:32.602723    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:32.604148 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:32.604121    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:32.604957 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:32.604936    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:10:34.322730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:34.314196    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:34.456731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:34.456252    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:35.225784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:35.225755    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:10:36.857238 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.857171    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:10:36.989683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.989605    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-envoy\") pod \"7451c167-4bdb-44dd-96ab-ed72782229d0\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") "
Oct 30 01:10:36.989930 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.989805    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-workload-certs\") pod \"7451c167-4bdb-44dd-96ab-ed72782229d0\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") "
Oct 30 01:10:36.990129 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.990102    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-workload-certs" pod "7451c167-4bdb-44dd-96ab-ed72782229d0" (UID: "7451c167-4bdb-44dd-96ab-ed72782229d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:36.990316 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.990295    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-token\" (UniqueName: \"kubernetes.io/projected/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-token\") pod \"7451c167-4bdb-44dd-96ab-ed72782229d0\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") "
Oct 30 01:10:36.990389 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.990372    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-workload-socket\") pod \"7451c167-4bdb-44dd-96ab-ed72782229d0\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") "
Oct 30 01:10:36.990688 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.990660    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-workload-socket" pod "7451c167-4bdb-44dd-96ab-ed72782229d0" (UID: "7451c167-4bdb-44dd-96ab-ed72782229d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:36.990860 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.990822    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-podinfo\") pod \"7451c167-4bdb-44dd-96ab-ed72782229d0\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") "
Oct 30 01:10:36.991205 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.991184    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-credential-socket\") pod \"7451c167-4bdb-44dd-96ab-ed72782229d0\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") "
Oct 30 01:10:36.991306 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.991250    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7451c167-4bdb-44dd-96ab-ed72782229d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7451c167-4bdb-44dd-96ab-ed72782229d0-istiod-ca-cert\") pod \"7451c167-4bdb-44dd-96ab-ed72782229d0\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") "
Oct 30 01:10:36.991373 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.991304    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-data\") pod \"7451c167-4bdb-44dd-96ab-ed72782229d0\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") "
Oct 30 01:10:36.991434 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.991400    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7451c167-4bdb-44dd-96ab-ed72782229d0-kube-api-access-sk7vl\" (UniqueName: \"kubernetes.io/projected/7451c167-4bdb-44dd-96ab-ed72782229d0-kube-api-access-sk7vl\") pod \"7451c167-4bdb-44dd-96ab-ed72782229d0\" (UID: \"7451c167-4bdb-44dd-96ab-ed72782229d0\") "
Oct 30 01:10:36.992596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.991415    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-credential-socket" pod "7451c167-4bdb-44dd-96ab-ed72782229d0" (UID: "7451c167-4bdb-44dd-96ab-ed72782229d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:36.992596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.991738    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7451c167-4bdb-44dd-96ab-ed72782229d0-istiod-ca-cert" pod "7451c167-4bdb-44dd-96ab-ed72782229d0" (UID: "7451c167-4bdb-44dd-96ab-ed72782229d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:36.992596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.991916    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-data" pod "7451c167-4bdb-44dd-96ab-ed72782229d0" (UID: "7451c167-4bdb-44dd-96ab-ed72782229d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:36.992596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.992244    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:36.992596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.992268    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:36.992596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.992282    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7451c167-4bdb-44dd-96ab-ed72782229d0-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:36.992596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.992296    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:36.992596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.992309    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:36.993105 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.992670    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-envoy" pod "7451c167-4bdb-44dd-96ab-ed72782229d0" (UID: "7451c167-4bdb-44dd-96ab-ed72782229d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:36.994661 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.994627    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7451c167-4bdb-44dd-96ab-ed72782229d0-kube-api-access-sk7vl" pod "7451c167-4bdb-44dd-96ab-ed72782229d0" (UID: "7451c167-4bdb-44dd-96ab-ed72782229d0"). InnerVolumeSpecName "kube-api-access-sk7vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:36.999276 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:36.999234    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-podinfo" pod "7451c167-4bdb-44dd-96ab-ed72782229d0" (UID: "7451c167-4bdb-44dd-96ab-ed72782229d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:10:37.002022 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:37.001993    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-token" pod "7451c167-4bdb-44dd-96ab-ed72782229d0" (UID: "7451c167-4bdb-44dd-96ab-ed72782229d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:37.093634 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:37.093596    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:37.093634 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:37.093627    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk7vl\" (UniqueName: \"kubernetes.io/projected/7451c167-4bdb-44dd-96ab-ed72782229d0-kube-api-access-sk7vl\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:37.093634 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:37.093639    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:37.093971 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:37.093648    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7451c167-4bdb-44dd-96ab-ed72782229d0-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:37.232818 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:37.232730    6099 generic.go:356] "Generic (PLEG): container finished" podID="7451c167-4bdb-44dd-96ab-ed72782229d0" containerID="c01b4f1bf335b3e4bcba294183c93c533fd0fb245fe7c0c52e3e29c843a245be" exitCode=0
Oct 30 01:10:37.232818 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:37.232777    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"
Oct 30 01:10:37.232818 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:37.232732    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8" event={"ID":"7451c167-4bdb-44dd-96ab-ed72782229d0","Type":"ContainerDied","Data":"c01b4f1bf335b3e4bcba294183c93c533fd0fb245fe7c0c52e3e29c843a245be"}
Oct 30 01:10:37.233092 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:37.232888    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8" event={"ID":"7451c167-4bdb-44dd-96ab-ed72782229d0","Type":"ContainerDied","Data":"8df930588df596700c6ed5446e74d9cf1b9625a8e6019f008e1fca6e9b17b9e2"}
Oct 30 01:10:37.233092 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:37.232911    6099 scope.go:122] "RemoveContainer" containerID="c01b4f1bf335b3e4bcba294183c93c533fd0fb245fe7c0c52e3e29c843a245be"
Oct 30 01:10:37.245027 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:37.245011    6099 scope.go:122] "RemoveContainer" containerID="c01b4f1bf335b3e4bcba294183c93c533fd0fb245fe7c0c52e3e29c843a245be"
Oct 30 01:10:37.245311 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:10:37.245291    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c01b4f1bf335b3e4bcba294183c93c533fd0fb245fe7c0c52e3e29c843a245be\": container with ID starting with c01b4f1bf335b3e4bcba294183c93c533fd0fb245fe7c0c52e3e29c843a245be not found: ID does not exist" containerID="c01b4f1bf335b3e4bcba294183c93c533fd0fb245fe7c0c52e3e29c843a245be"
Oct 30 01:10:37.245403 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:37.245318    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c01b4f1bf335b3e4bcba294183c93c533fd0fb245fe7c0c52e3e29c843a245be"} err="failed to get container status \"c01b4f1bf335b3e4bcba294183c93c533fd0fb245fe7c0c52e3e29c843a245be\": rpc error: code = NotFound desc = could not find container \"c01b4f1bf335b3e4bcba294183c93c533fd0fb245fe7c0c52e3e29c843a245be\": container with ID starting with c01b4f1bf335b3e4bcba294183c93c533fd0fb245fe7c0c52e3e29c843a245be not found: ID does not exist"
Oct 30 01:10:37.288493 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:37.288462    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"]
Oct 30 01:10:37.299841 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:37.299818    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1da7b2-openshift-default-6997bcc7b7-mflx8"]
Oct 30 01:10:38.166334 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:38.166306    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"
Oct 30 01:10:38.266652 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:38.266625    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7451c167-4bdb-44dd-96ab-ed72782229d0" path="/var/lib/kubelet/pods/7451c167-4bdb-44dd-96ab-ed72782229d0/volumes"
Oct 30 01:10:46.630800 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:10:46.630764    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:10:48.432335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:48.432301    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"]
Oct 30 01:10:48.432820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:48.432571    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb" podUID="7c7187ea-003c-49c3-a37f-1f4103649d1d" containerName="maas-api" containerID="cri-o://92b844dff7c991e808250d08c2c1bd3dae2f250595701a0cdc6559b840a30dc2" gracePeriod=30
Oct 30 01:10:48.797458 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:48.797428    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:10:48.915571 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:48.915534    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c7187ea-003c-49c3-a37f-1f4103649d1d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7187ea-003c-49c3-a37f-1f4103649d1d-maas-api-tls\") pod \"7c7187ea-003c-49c3-a37f-1f4103649d1d\" (UID: \"7c7187ea-003c-49c3-a37f-1f4103649d1d\") "
Oct 30 01:10:48.917239 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:48.917192    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7187ea-003c-49c3-a37f-1f4103649d1d-openshift-service-ca" pod "7c7187ea-003c-49c3-a37f-1f4103649d1d" (UID: "7c7187ea-003c-49c3-a37f-1f4103649d1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:48.917392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:48.917375    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c7187ea-003c-49c3-a37f-1f4103649d1d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7187ea-003c-49c3-a37f-1f4103649d1d-openshift-service-ca\") pod \"7c7187ea-003c-49c3-a37f-1f4103649d1d\" (UID: \"7c7187ea-003c-49c3-a37f-1f4103649d1d\") "
Oct 30 01:10:48.917615 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:48.917600    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7187ea-003c-49c3-a37f-1f4103649d1d-kube-api-access-frwp2\" (UniqueName: \"kubernetes.io/projected/7c7187ea-003c-49c3-a37f-1f4103649d1d-kube-api-access-frwp2\") pod \"7c7187ea-003c-49c3-a37f-1f4103649d1d\" (UID: \"7c7187ea-003c-49c3-a37f-1f4103649d1d\") "
Oct 30 01:10:48.918210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:48.918191    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c7187ea-003c-49c3-a37f-1f4103649d1d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7187ea-003c-49c3-a37f-1f4103649d1d-metrics-certs\") pod \"7c7187ea-003c-49c3-a37f-1f4103649d1d\" (UID: \"7c7187ea-003c-49c3-a37f-1f4103649d1d\") "
Oct 30 01:10:48.919486 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:48.919364    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7187ea-003c-49c3-a37f-1f4103649d1d-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:48.930245 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:48.930189    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7187ea-003c-49c3-a37f-1f4103649d1d-maas-api-tls" pod "7c7187ea-003c-49c3-a37f-1f4103649d1d" (UID: "7c7187ea-003c-49c3-a37f-1f4103649d1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:10:48.930884 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:48.930846    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7187ea-003c-49c3-a37f-1f4103649d1d-metrics-certs" pod "7c7187ea-003c-49c3-a37f-1f4103649d1d" (UID: "7c7187ea-003c-49c3-a37f-1f4103649d1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:10:48.935341 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:48.935305    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7187ea-003c-49c3-a37f-1f4103649d1d-kube-api-access-frwp2" pod "7c7187ea-003c-49c3-a37f-1f4103649d1d" (UID: "7c7187ea-003c-49c3-a37f-1f4103649d1d"). InnerVolumeSpecName "kube-api-access-frwp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:49.021183 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:49.021150    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7187ea-003c-49c3-a37f-1f4103649d1d-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:49.021183 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:49.021176    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7187ea-003c-49c3-a37f-1f4103649d1d-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:49.021183 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:49.021187    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frwp2\" (UniqueName: \"kubernetes.io/projected/7c7187ea-003c-49c3-a37f-1f4103649d1d-kube-api-access-frwp2\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:10:49.347621 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:49.347589    6099 generic.go:356] "Generic (PLEG): container finished" podID="7c7187ea-003c-49c3-a37f-1f4103649d1d" containerID="92b844dff7c991e808250d08c2c1bd3dae2f250595701a0cdc6559b840a30dc2" exitCode=0
Oct 30 01:10:49.347872 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:49.347636    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"
Oct 30 01:10:49.347872 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:49.347674    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb" event={"ID":"7c7187ea-003c-49c3-a37f-1f4103649d1d","Type":"ContainerDied","Data":"92b844dff7c991e808250d08c2c1bd3dae2f250595701a0cdc6559b840a30dc2"}
Oct 30 01:10:49.347872 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:49.347769    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb" event={"ID":"7c7187ea-003c-49c3-a37f-1f4103649d1d","Type":"ContainerDied","Data":"28e2f3a614187af37749ec39532ba077f749ea6feef3e5875bac7cb723a7c9be"}
Oct 30 01:10:49.347872 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:49.347807    6099 scope.go:122] "RemoveContainer" containerID="92b844dff7c991e808250d08c2c1bd3dae2f250595701a0cdc6559b840a30dc2"
Oct 30 01:10:49.359818 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:49.359801    6099 scope.go:122] "RemoveContainer" containerID="92b844dff7c991e808250d08c2c1bd3dae2f250595701a0cdc6559b840a30dc2"
Oct 30 01:10:49.360078 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:10:49.360056    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b844dff7c991e808250d08c2c1bd3dae2f250595701a0cdc6559b840a30dc2\": container with ID starting with 92b844dff7c991e808250d08c2c1bd3dae2f250595701a0cdc6559b840a30dc2 not found: ID does not exist" containerID="92b844dff7c991e808250d08c2c1bd3dae2f250595701a0cdc6559b840a30dc2"
Oct 30 01:10:49.360127 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:49.360084    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b844dff7c991e808250d08c2c1bd3dae2f250595701a0cdc6559b840a30dc2"} err="failed to get container status \"92b844dff7c991e808250d08c2c1bd3dae2f250595701a0cdc6559b840a30dc2\": rpc error: code = NotFound desc = could not find container \"92b844dff7c991e808250d08c2c1bd3dae2f250595701a0cdc6559b840a30dc2\": container with ID starting with 92b844dff7c991e808250d08c2c1bd3dae2f250595701a0cdc6559b840a30dc2 not found: ID does not exist"
Oct 30 01:10:49.402028 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:49.402001    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"]
Oct 30 01:10:49.429175 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:49.429139    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f96ede-b79fdd74f-m6vxb"]
Oct 30 01:10:49.501340 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:49.501306    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc"]
Oct 30 01:10:49.505263 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:49.505238    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f96ede-4s4cc"]
Oct 30 01:10:50.264740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:50.264687    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aba30ff-23fb-4bbb-a858-562cd2a67ac2" path="/var/lib/kubelet/pods/6aba30ff-23fb-4bbb-a858-562cd2a67ac2/volumes"
Oct 30 01:10:50.265184 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:50.265169    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7187ea-003c-49c3-a37f-1f4103649d1d" path="/var/lib/kubelet/pods/7c7187ea-003c-49c3-a37f-1f4103649d1d/volumes"
Oct 30 01:10:51.381716 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.381668    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"]
Oct 30 01:10:51.382390 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.382375    6099 cpu_manager.go:414] "containerMap: removing container" podUID="7c7187ea-003c-49c3-a37f-1f4103649d1d" containerName="maas-api"
Oct 30 01:10:51.382445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.382391    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7187ea-003c-49c3-a37f-1f4103649d1d" containerName="maas-api"
Oct 30 01:10:51.382445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.382440    6099 cpu_manager.go:414] "containerMap: removing container" podUID="7451c167-4bdb-44dd-96ab-ed72782229d0" containerName="istio-proxy"
Oct 30 01:10:51.382510 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.382446    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7451c167-4bdb-44dd-96ab-ed72782229d0" containerName="istio-proxy"
Oct 30 01:10:51.382571 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.382562    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7451c167-4bdb-44dd-96ab-ed72782229d0" containerName="istio-proxy"
Oct 30 01:10:51.382609 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.382576    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7187ea-003c-49c3-a37f-1f4103649d1d" containerName="maas-api"
Oct 30 01:10:51.388157 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.388131    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"
Oct 30 01:10:51.421968 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.421937    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"]
Oct 30 01:10:51.448333 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.447368    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-d01638-serving-cert\""
Oct 30 01:10:51.463672 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.463632    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb\" (UID: \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"
Oct 30 01:10:51.463874 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.463769    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb\" (UID: \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"
Oct 30 01:10:51.463874 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.463819    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crxzn\" (UniqueName: \"kubernetes.io/projected/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-kube-api-access-crxzn\") pod \"maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb\" (UID: \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"
Oct 30 01:10:51.464016 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.463949    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb\" (UID: \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"
Oct 30 01:10:51.565568 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.565532    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb\" (UID: \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"
Oct 30 01:10:51.565777 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.565645    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb\" (UID: \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"
Oct 30 01:10:51.565777 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.565691    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crxzn\" (UniqueName: \"kubernetes.io/projected/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-kube-api-access-crxzn\") pod \"maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb\" (UID: \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"
Oct 30 01:10:51.565904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.565824    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb\" (UID: \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"
Oct 30 01:10:51.566374 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.566349    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb\" (UID: \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"
Oct 30 01:10:51.568138 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.568114    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb\" (UID: \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"
Oct 30 01:10:51.568249 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.568147    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb\" (UID: \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"
Oct 30 01:10:51.596057 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.596027    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crxzn\" (UniqueName: \"kubernetes.io/projected/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-kube-api-access-crxzn\") pod \"maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb\" (UID: \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"
Oct 30 01:10:51.703253 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:51.703169    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"
Oct 30 01:10:52.014319 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:10:52.014287    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7116e8fb_516c_486d_ae70_9f8cf1ad0d2a.slice/crio-c26c049674f74c70431bafbcf6f51e26a5e133b92d2d6548e8422e44b85a5d45 WatchSource:0}: Error finding container c26c049674f74c70431bafbcf6f51e26a5e133b92d2d6548e8422e44b85a5d45: Status 404 returned error can't find the container with id c26c049674f74c70431bafbcf6f51e26a5e133b92d2d6548e8422e44b85a5d45
Oct 30 01:10:52.015322 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:52.015295    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"]
Oct 30 01:10:52.376172 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:52.376144    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb" event={"ID":"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a","Type":"ContainerStarted","Data":"c26c049674f74c70431bafbcf6f51e26a5e133b92d2d6548e8422e44b85a5d45"}
Oct 30 01:10:53.385361 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:53.385320    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb" event={"ID":"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a","Type":"ContainerStarted","Data":"91c4141c30a6dd725f849a88268adaf7c87bb51cf995ad61ac1c83d5827892dc"}
Oct 30 01:10:53.385786 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:53.385396    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"
Oct 30 01:10:53.428925 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:53.428878    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb" podStartSLOduration=2.10513615 podStartE2EDuration="2.428865041s" totalImagesPullingTime="323.728891ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:53.426406636 +0000 UTC m=+2147.283314390" watchObservedRunningTime="2026-10-30 01:10:53.428865041 +0000 UTC m=+2147.285772796"
Oct 30 01:10:56.765218 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:56.765182    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"]
Oct 30 01:10:56.765734 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:56.765675    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76" podUID="edf31a9e-34ff-428b-9c36-4189119697d7" containerName="istio-proxy" containerID="cri-o://acd4064f9bfa8deb15f1561659227bddcf7d83b36d3eb61f22ed9cdcf9259449" gracePeriod=30
Oct 30 01:10:57.534456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.534418    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"]
Oct 30 01:10:57.540096 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.540066    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.560382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.560350    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"]
Oct 30 01:10:57.566000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.565972    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-62f5d1\"/\"auth-test-model-62f5d1-kserve-self-signed-certs\""
Oct 30 01:10:57.566655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.566633    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-62f5d1\"/\"default-dockercfg-thjwf\""
Oct 30 01:10:57.566727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.566633    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-62f5d1\"/\"kube-root-ca.crt\""
Oct 30 01:10:57.566910 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.566894    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-62f5d1\"/\"openshift-service-ca.crt\""
Oct 30 01:10:57.683623 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.683588    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-tmp-dir\") pod \"auth-test-model-62f5d1-kserve-74557c65f7-6nzhb\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") " pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.683831 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.683637    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-tls-certs\") pod \"auth-test-model-62f5d1-kserve-74557c65f7-6nzhb\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") " pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.683831 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.683820    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-home\") pod \"auth-test-model-62f5d1-kserve-74557c65f7-6nzhb\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") " pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.683954 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.683877    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt9vs\" (UniqueName: \"kubernetes.io/projected/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-kube-api-access-bt9vs\") pod \"auth-test-model-62f5d1-kserve-74557c65f7-6nzhb\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") " pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.684050 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.684028    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-model-cache\") pod \"auth-test-model-62f5d1-kserve-74557c65f7-6nzhb\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") " pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.684109 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.684076    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-dshm\") pod \"auth-test-model-62f5d1-kserve-74557c65f7-6nzhb\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") " pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.808784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.788684    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-tmp-dir\") pod \"auth-test-model-62f5d1-kserve-74557c65f7-6nzhb\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") " pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.808784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.786918    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-tmp-dir\") pod \"auth-test-model-62f5d1-kserve-74557c65f7-6nzhb\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") " pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.808784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.788818    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-tls-certs\") pod \"auth-test-model-62f5d1-kserve-74557c65f7-6nzhb\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") " pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.808784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.797216    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-home\") pod \"auth-test-model-62f5d1-kserve-74557c65f7-6nzhb\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") " pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.808784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.797312    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt9vs\" (UniqueName: \"kubernetes.io/projected/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-kube-api-access-bt9vs\") pod \"auth-test-model-62f5d1-kserve-74557c65f7-6nzhb\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") " pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.808784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.797474    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-model-cache\") pod \"auth-test-model-62f5d1-kserve-74557c65f7-6nzhb\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") " pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.808784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.797568    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-dshm\") pod \"auth-test-model-62f5d1-kserve-74557c65f7-6nzhb\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") " pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.808784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.798503    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-home\") pod \"auth-test-model-62f5d1-kserve-74557c65f7-6nzhb\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") " pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.808784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.799462    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-model-cache\") pod \"auth-test-model-62f5d1-kserve-74557c65f7-6nzhb\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") " pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.813727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.810848    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-tls-certs\") pod \"auth-test-model-62f5d1-kserve-74557c65f7-6nzhb\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") " pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.827756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.825196    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-dshm\") pod \"auth-test-model-62f5d1-kserve-74557c65f7-6nzhb\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") " pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.830734 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.829616    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt9vs\" (UniqueName: \"kubernetes.io/projected/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-kube-api-access-bt9vs\") pod \"auth-test-model-62f5d1-kserve-74557c65f7-6nzhb\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") " pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:57.859635 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:57.859598    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:10:58.020643 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.020612    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"]
Oct 30 01:10:58.026920 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.026897    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.050594 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.050564    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-320045-openshift-default-dockercfg-cbfcp\""
Oct 30 01:10:58.052716 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.052677    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"]
Oct 30 01:10:58.114465 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.114391    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.114465 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.114442    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.114465 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.114465    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.114741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.114579    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.114741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.114689    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.114855 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.114786    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp8wp\" (UniqueName: \"kubernetes.io/projected/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-kube-api-access-cp8wp\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.114855 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.114824    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.114977 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.114880    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.114977 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.114910    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.233726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.228189    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.233726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.228272    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp8wp\" (UniqueName: \"kubernetes.io/projected/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-kube-api-access-cp8wp\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.233726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.228313    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.233726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.228348    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.233726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.228384    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.233726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.228598    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.233726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.228661    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.233726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.228690    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.233726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.228775    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.233726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.229374    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.233726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.229621    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.233726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.230559    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.233726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.231003    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.233726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.232846    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.247730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.247338    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.251726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.249070    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.285811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.283334    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.285811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.285322    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp8wp\" (UniqueName: \"kubernetes.io/projected/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-kube-api-access-cp8wp\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.311505 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:10:58.311461    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30fbfcf0_f18a_4949_92d8_fbbb726ed62d.slice/crio-121115c73ee513e49aa2e7486cda356b0169fb392e3617c8a7a5c50daa533b87 WatchSource:0}: Error finding container 121115c73ee513e49aa2e7486cda356b0169fb392e3617c8a7a5c50daa533b87: Status 404 returned error can't find the container with id 121115c73ee513e49aa2e7486cda356b0169fb392e3617c8a7a5c50daa533b87
Oct 30 01:10:58.354729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.350387    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"]
Oct 30 01:10:58.362526 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.360065    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:10:58.427828 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.427800    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb" event={"ID":"30fbfcf0-f18a-4949-92d8-fbbb726ed62d","Type":"ContainerStarted","Data":"121115c73ee513e49aa2e7486cda356b0169fb392e3617c8a7a5c50daa533b87"}
Oct 30 01:10:58.728985 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:10:58.728952    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc28ff31_b61a_4d8e_9ff3_361345eb9c06.slice/crio-9463b6c0af14f2cd3fc7fe52febb8264edc21f80d36e38d68fdb3069a9cf22d8 WatchSource:0}: Error finding container 9463b6c0af14f2cd3fc7fe52febb8264edc21f80d36e38d68fdb3069a9cf22d8: Status 404 returned error can't find the container with id 9463b6c0af14f2cd3fc7fe52febb8264edc21f80d36e38d68fdb3069a9cf22d8
Oct 30 01:10:58.729537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.729514    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"]
Oct 30 01:10:58.733759 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.732857    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:10:58.733759 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.733022    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:10:58.733759 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:58.733110    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:10:59.399371 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:59.399338    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"
Oct 30 01:10:59.438606 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:59.438568    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7" event={"ID":"cc28ff31-b61a-4d8e-9ff3-361345eb9c06","Type":"ContainerStarted","Data":"023512b0f3b8ca26813352bab4f7f74e5b2c877a51d9d733976ef77ae1fd5f58"}
Oct 30 01:10:59.438818 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:59.438632    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7" event={"ID":"cc28ff31-b61a-4d8e-9ff3-361345eb9c06","Type":"ContainerStarted","Data":"9463b6c0af14f2cd3fc7fe52febb8264edc21f80d36e38d68fdb3069a9cf22d8"}
Oct 30 01:10:59.440429 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:59.440399    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb" event={"ID":"30fbfcf0-f18a-4949-92d8-fbbb726ed62d","Type":"ContainerStarted","Data":"f50edf569e1c2a0f9e3f3376c29bb5374e252c00ae6839551dc0ced0a59bfe16"}
Oct 30 01:10:59.487282 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:59.487221    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7" podStartSLOduration=2.487203666 podStartE2EDuration="2.487203666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:59.485145957 +0000 UTC m=+2153.342053711" watchObservedRunningTime="2026-10-30 01:10:59.487203666 +0000 UTC m=+2153.344111423"
Oct 30 01:10:59.529481 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:59.529431    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb" podStartSLOduration=2.529417675 podStartE2EDuration="2.529417675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:59.526156445 +0000 UTC m=+2153.383064200" watchObservedRunningTime="2026-10-30 01:10:59.529417675 +0000 UTC m=+2153.386325429"
Oct 30 01:10:59.860051 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:10:59.860016    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:11:00.358446 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.358409    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"]
Oct 30 01:11:00.363399 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.363375    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:11:00.363606 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.363493    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.368478 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.368455    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:11:00.382862 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.382834    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"]
Oct 30 01:11:00.389673 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.389639    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-ea2547\"/\"openshift-service-ca.crt\""
Oct 30 01:11:00.389673 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.389648    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-ea2547\"/\"kube-root-ca.crt\""
Oct 30 01:11:00.392984 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.392964    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-ea2547\"/\"sub-test-model-ea2547-kserve-self-signed-certs\""
Oct 30 01:11:00.393090 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.393019    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-ea2547\"/\"default-dockercfg-6cflv\""
Oct 30 01:11:00.449237 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.449208    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:11:00.449966 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.449947    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:11:00.470894 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.470871    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-home\") pod \"sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") " pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.471014 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.470924    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-tmp-dir\") pod \"sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") " pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.471058 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.471021    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-dshm\") pod \"sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") " pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.471114 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.471063    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-model-cache\") pod \"sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") " pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.471180 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.471165    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4p2s\" (UniqueName: \"kubernetes.io/projected/5de95229-e8f1-483d-8540-15e1d33182ea-kube-api-access-v4p2s\") pod \"sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") " pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.471272 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.471259    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de95229-e8f1-483d-8540-15e1d33182ea-tls-certs\") pod \"sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") " pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.572924 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.572887    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-home\") pod \"sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") " pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.573128 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.572956    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-tmp-dir\") pod \"sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") " pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.573128 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.573010    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-dshm\") pod \"sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") " pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.573128 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.573055    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-model-cache\") pod \"sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") " pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.573356 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.573285    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4p2s\" (UniqueName: \"kubernetes.io/projected/5de95229-e8f1-483d-8540-15e1d33182ea-kube-api-access-v4p2s\") pod \"sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") " pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.573428 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.573356    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-home\") pod \"sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") " pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.573428 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.573408    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-tmp-dir\") pod \"sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") " pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.573540 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.573451    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de95229-e8f1-483d-8540-15e1d33182ea-tls-certs\") pod \"sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") " pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.574162 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.574132    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-model-cache\") pod \"sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") " pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.575955 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.575934    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-dshm\") pod \"sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") " pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.576174 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.576153    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de95229-e8f1-483d-8540-15e1d33182ea-tls-certs\") pod \"sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") " pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.596210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.596188    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4p2s\" (UniqueName: \"kubernetes.io/projected/5de95229-e8f1-483d-8540-15e1d33182ea-kube-api-access-v4p2s\") pod \"sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") " pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:00.688403 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:00.687996    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:01.040053 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:11:01.040022    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5de95229_e8f1_483d_8540_15e1d33182ea.slice/crio-1bbd12531bed2a5babdebc6e90a13ea499c21f6feb37739718cdc72b6bd5c747 WatchSource:0}: Error finding container 1bbd12531bed2a5babdebc6e90a13ea499c21f6feb37739718cdc72b6bd5c747: Status 404 returned error can't find the container with id 1bbd12531bed2a5babdebc6e90a13ea499c21f6feb37739718cdc72b6bd5c747
Oct 30 01:11:01.040937 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:01.040915    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"]
Oct 30 01:11:01.460784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:01.460749    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk" event={"ID":"5de95229-e8f1-483d-8540-15e1d33182ea","Type":"ContainerStarted","Data":"7e4ae806feb805b6690342d78d76d18f9bf289eea700aef79f666a04c799069c"}
Oct 30 01:11:01.460784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:01.460798    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk" event={"ID":"5de95229-e8f1-483d-8540-15e1d33182ea","Type":"ContainerStarted","Data":"1bbd12531bed2a5babdebc6e90a13ea499c21f6feb37739718cdc72b6bd5c747"}
Oct 30 01:11:01.501752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:01.501692    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk" podStartSLOduration=1.501677755 podStartE2EDuration="1.501677755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:01.500044552 +0000 UTC m=+2155.356952306" watchObservedRunningTime="2026-10-30 01:11:01.501677755 +0000 UTC m=+2155.358585504"
Oct 30 01:11:01.689852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:01.689821    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:02.062295 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.062262    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:11:02.104545 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.104519    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf31a9e-34ff-428b-9c36-4189119697d7-istio-token\" (UniqueName: \"kubernetes.io/projected/edf31a9e-34ff-428b-9c36-4189119697d7-istio-token\") pod \"edf31a9e-34ff-428b-9c36-4189119697d7\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") "
Oct 30 01:11:02.104689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.104608    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-istio-data\") pod \"edf31a9e-34ff-428b-9c36-4189119697d7\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") "
Oct 30 01:11:02.104689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.104635    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edf31a9e-34ff-428b-9c36-4189119697d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edf31a9e-34ff-428b-9c36-4189119697d7-istiod-ca-cert\") pod \"edf31a9e-34ff-428b-9c36-4189119697d7\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") "
Oct 30 01:11:02.104689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.104673    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-istio-envoy\") pod \"edf31a9e-34ff-428b-9c36-4189119697d7\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") "
Oct 30 01:11:02.104857 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.104756    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-workload-certs\") pod \"edf31a9e-34ff-428b-9c36-4189119697d7\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") "
Oct 30 01:11:02.104857 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.104795    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-workload-socket\") pod \"edf31a9e-34ff-428b-9c36-4189119697d7\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") "
Oct 30 01:11:02.104857 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.104853    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edf31a9e-34ff-428b-9c36-4189119697d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edf31a9e-34ff-428b-9c36-4189119697d7-istio-podinfo\") pod \"edf31a9e-34ff-428b-9c36-4189119697d7\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") "
Oct 30 01:11:02.105065 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.104903    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf31a9e-34ff-428b-9c36-4189119697d7-kube-api-access-pd2l5\" (UniqueName: \"kubernetes.io/projected/edf31a9e-34ff-428b-9c36-4189119697d7-kube-api-access-pd2l5\") pod \"edf31a9e-34ff-428b-9c36-4189119697d7\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") "
Oct 30 01:11:02.105065 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.104952    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-istio-data" pod "edf31a9e-34ff-428b-9c36-4189119697d7" (UID: "edf31a9e-34ff-428b-9c36-4189119697d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:02.105183 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.105084    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-credential-socket\") pod \"edf31a9e-34ff-428b-9c36-4189119697d7\" (UID: \"edf31a9e-34ff-428b-9c36-4189119697d7\") "
Oct 30 01:11:02.105183 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.105106    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-workload-certs" pod "edf31a9e-34ff-428b-9c36-4189119697d7" (UID: "edf31a9e-34ff-428b-9c36-4189119697d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:02.105298 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.105189    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-workload-socket" pod "edf31a9e-34ff-428b-9c36-4189119697d7" (UID: "edf31a9e-34ff-428b-9c36-4189119697d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:02.105298 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.105116    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf31a9e-34ff-428b-9c36-4189119697d7-istiod-ca-cert" pod "edf31a9e-34ff-428b-9c36-4189119697d7" (UID: "edf31a9e-34ff-428b-9c36-4189119697d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:02.105492 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.105468    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-credential-socket" pod "edf31a9e-34ff-428b-9c36-4189119697d7" (UID: "edf31a9e-34ff-428b-9c36-4189119697d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:02.105956 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.105931    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:11:02.106056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.105960    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:11:02.106056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.105974    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edf31a9e-34ff-428b-9c36-4189119697d7-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:11:02.106056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.105988    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:11:02.106056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.106001    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:11:02.107138 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.107110    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf31a9e-34ff-428b-9c36-4189119697d7-istio-token" pod "edf31a9e-34ff-428b-9c36-4189119697d7" (UID: "edf31a9e-34ff-428b-9c36-4189119697d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:02.107285 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.107266    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf31a9e-34ff-428b-9c36-4189119697d7-kube-api-access-pd2l5" pod "edf31a9e-34ff-428b-9c36-4189119697d7" (UID: "edf31a9e-34ff-428b-9c36-4189119697d7"). InnerVolumeSpecName "kube-api-access-pd2l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:02.107359 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.107338    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-istio-envoy" pod "edf31a9e-34ff-428b-9c36-4189119697d7" (UID: "edf31a9e-34ff-428b-9c36-4189119697d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:02.107401 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.107383    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edf31a9e-34ff-428b-9c36-4189119697d7-istio-podinfo" pod "edf31a9e-34ff-428b-9c36-4189119697d7" (UID: "edf31a9e-34ff-428b-9c36-4189119697d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:11:02.207326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.207294    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edf31a9e-34ff-428b-9c36-4189119697d7-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:11:02.207326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.207321    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edf31a9e-34ff-428b-9c36-4189119697d7-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:11:02.207326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.207331    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd2l5\" (UniqueName: \"kubernetes.io/projected/edf31a9e-34ff-428b-9c36-4189119697d7-kube-api-access-pd2l5\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:11:02.207566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.207340    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edf31a9e-34ff-428b-9c36-4189119697d7-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:11:02.474339 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.474303    6099 generic.go:356] "Generic (PLEG): container finished" podID="edf31a9e-34ff-428b-9c36-4189119697d7" containerID="acd4064f9bfa8deb15f1561659227bddcf7d83b36d3eb61f22ed9cdcf9259449" exitCode=0
Oct 30 01:11:02.474339 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.474338    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"
Oct 30 01:11:02.474945 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.474374    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76" event={"ID":"edf31a9e-34ff-428b-9c36-4189119697d7","Type":"ContainerDied","Data":"acd4064f9bfa8deb15f1561659227bddcf7d83b36d3eb61f22ed9cdcf9259449"}
Oct 30 01:11:02.474945 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.474444    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76" event={"ID":"edf31a9e-34ff-428b-9c36-4189119697d7","Type":"ContainerDied","Data":"988816cce779e3973d8ed31f3270b9765d6cac875fb97a86fcb7ea6314262c7b"}
Oct 30 01:11:02.474945 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.474478    6099 scope.go:122] "RemoveContainer" containerID="acd4064f9bfa8deb15f1561659227bddcf7d83b36d3eb61f22ed9cdcf9259449"
Oct 30 01:11:02.492238 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.492217    6099 scope.go:122] "RemoveContainer" containerID="acd4064f9bfa8deb15f1561659227bddcf7d83b36d3eb61f22ed9cdcf9259449"
Oct 30 01:11:02.492575 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:11:02.492550    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd4064f9bfa8deb15f1561659227bddcf7d83b36d3eb61f22ed9cdcf9259449\": container with ID starting with acd4064f9bfa8deb15f1561659227bddcf7d83b36d3eb61f22ed9cdcf9259449 not found: ID does not exist" containerID="acd4064f9bfa8deb15f1561659227bddcf7d83b36d3eb61f22ed9cdcf9259449"
Oct 30 01:11:02.492642 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.492584    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd4064f9bfa8deb15f1561659227bddcf7d83b36d3eb61f22ed9cdcf9259449"} err="failed to get container status \"acd4064f9bfa8deb15f1561659227bddcf7d83b36d3eb61f22ed9cdcf9259449\": rpc error: code = NotFound desc = could not find container \"acd4064f9bfa8deb15f1561659227bddcf7d83b36d3eb61f22ed9cdcf9259449\": container with ID starting with acd4064f9bfa8deb15f1561659227bddcf7d83b36d3eb61f22ed9cdcf9259449 not found: ID does not exist"
Oct 30 01:11:02.532384 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.532355    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"]
Oct 30 01:11:02.538032 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:02.538006    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f96ede-openshift-default-684654576c-whs76"]
Oct 30 01:11:03.913833 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:03.913796    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p"]
Oct 30 01:11:03.914751 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:03.914693    6099 cpu_manager.go:414] "containerMap: removing container" podUID="edf31a9e-34ff-428b-9c36-4189119697d7" containerName="istio-proxy"
Oct 30 01:11:03.914751 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:03.914731    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf31a9e-34ff-428b-9c36-4189119697d7" containerName="istio-proxy"
Oct 30 01:11:03.915011 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:03.914994    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="edf31a9e-34ff-428b-9c36-4189119697d7" containerName="istio-proxy"
Oct 30 01:11:03.920142 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:03.920121    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p"
Oct 30 01:11:03.933449 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:03.933416    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed5048f-9425-41d3-8d13-7b149709bf3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p\" (UID: \"7ed5048f-9425-41d3-8d13-7b149709bf3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p"
Oct 30 01:11:03.933781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:03.933747    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh97l\" (UniqueName: \"kubernetes.io/projected/7ed5048f-9425-41d3-8d13-7b149709bf3a-kube-api-access-wh97l\") pod \"payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p\" (UID: \"7ed5048f-9425-41d3-8d13-7b149709bf3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p"
Oct 30 01:11:03.938451 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:03.938428    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p"]
Oct 30 01:11:03.944319 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:03.944287    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-320045\""
Oct 30 01:11:03.944421 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:03.944371    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-320045-dockercfg-q2dtr\""
Oct 30 01:11:04.115922 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:04.115802    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed5048f-9425-41d3-8d13-7b149709bf3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p\" (UID: \"7ed5048f-9425-41d3-8d13-7b149709bf3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p"
Oct 30 01:11:04.116468 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:04.116429    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh97l\" (UniqueName: \"kubernetes.io/projected/7ed5048f-9425-41d3-8d13-7b149709bf3a-kube-api-access-wh97l\") pod \"payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p\" (UID: \"7ed5048f-9425-41d3-8d13-7b149709bf3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p"
Oct 30 01:11:04.121618 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:04.117368    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed5048f-9425-41d3-8d13-7b149709bf3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p\" (UID: \"7ed5048f-9425-41d3-8d13-7b149709bf3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p"
Oct 30 01:11:04.124185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:04.124164    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4"]
Oct 30 01:11:04.136765 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:04.134945    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4"]
Oct 30 01:11:04.136765 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:04.135060    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4"
Oct 30 01:11:04.155380 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:04.149363    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh97l\" (UniqueName: \"kubernetes.io/projected/7ed5048f-9425-41d3-8d13-7b149709bf3a-kube-api-access-wh97l\") pod \"payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p\" (UID: \"7ed5048f-9425-41d3-8d13-7b149709bf3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p"
Oct 30 01:11:04.237415 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:04.235236    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p"
Oct 30 01:11:04.267623 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:04.267590    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edf31a9e-34ff-428b-9c36-4189119697d7" path="/var/lib/kubelet/pods/edf31a9e-34ff-428b-9c36-4189119697d7/volumes"
Oct 30 01:11:04.321078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:04.320987    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24hm2\" (UniqueName: \"kubernetes.io/projected/d3bfb222-a8f8-47bc-a3cd-a62716211f0e-kube-api-access-24hm2\") pod \"payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4\" (UID: \"d3bfb222-a8f8-47bc-a3cd-a62716211f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4"
Oct 30 01:11:04.321347 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:04.321321    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3bfb222-a8f8-47bc-a3cd-a62716211f0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4\" (UID: \"d3bfb222-a8f8-47bc-a3cd-a62716211f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4"
Oct 30 01:11:04.423421 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:04.423383    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24hm2\" (UniqueName: \"kubernetes.io/projected/d3bfb222-a8f8-47bc-a3cd-a62716211f0e-kube-api-access-24hm2\") pod \"payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4\" (UID: \"d3bfb222-a8f8-47bc-a3cd-a62716211f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4"
Oct 30 01:11:04.423770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:04.423748    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3bfb222-a8f8-47bc-a3cd-a62716211f0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4\" (UID: \"d3bfb222-a8f8-47bc-a3cd-a62716211f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4"
Oct 30 01:11:04.424390 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:04.424364    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3bfb222-a8f8-47bc-a3cd-a62716211f0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4\" (UID: \"d3bfb222-a8f8-47bc-a3cd-a62716211f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4"
Oct 30 01:11:04.446633 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:04.446612    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24hm2\" (UniqueName: \"kubernetes.io/projected/d3bfb222-a8f8-47bc-a3cd-a62716211f0e-kube-api-access-24hm2\") pod \"payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4\" (UID: \"d3bfb222-a8f8-47bc-a3cd-a62716211f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4"
Oct 30 01:11:04.517329 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:04.517255    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4"
Oct 30 01:11:04.551622 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:11:04.551592    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed5048f_9425_41d3_8d13_7b149709bf3a.slice/crio-5c7b746397bf9d52af3deefe22ffe616f26a72cf9d253b42e9a83dd44a097a4a WatchSource:0}: Error finding container 5c7b746397bf9d52af3deefe22ffe616f26a72cf9d253b42e9a83dd44a097a4a: Status 404 returned error can't find the container with id 5c7b746397bf9d52af3deefe22ffe616f26a72cf9d253b42e9a83dd44a097a4a
Oct 30 01:11:04.552349 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:04.552322    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p"]
Oct 30 01:11:04.835666 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:11:04.835637    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3bfb222_a8f8_47bc_a3cd_a62716211f0e.slice/crio-1613f07a0c36d13a3112b609f40ea87687cbae01303be2b29fd584d702785340 WatchSource:0}: Error finding container 1613f07a0c36d13a3112b609f40ea87687cbae01303be2b29fd584d702785340: Status 404 returned error can't find the container with id 1613f07a0c36d13a3112b609f40ea87687cbae01303be2b29fd584d702785340
Oct 30 01:11:04.835916 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:04.835898    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4"]
Oct 30 01:11:05.503411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:05.503365    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p" event={"ID":"7ed5048f-9425-41d3-8d13-7b149709bf3a","Type":"ContainerStarted","Data":"7ab173004a8dbd025bd04cd462f30a89d5e467fca6fefa00af92004de97159a1"}
Oct 30 01:11:05.503921 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:05.503437    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p" event={"ID":"7ed5048f-9425-41d3-8d13-7b149709bf3a","Type":"ContainerStarted","Data":"5c7b746397bf9d52af3deefe22ffe616f26a72cf9d253b42e9a83dd44a097a4a"}
Oct 30 01:11:05.503921 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:05.503466    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p"
Oct 30 01:11:05.505408 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:05.505375    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4" event={"ID":"d3bfb222-a8f8-47bc-a3cd-a62716211f0e","Type":"ContainerStarted","Data":"5b9a5538abb8f89f82732a9e0584c1f58642ae673a1c6a509b540d16d0b1c88b"}
Oct 30 01:11:05.505540 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:05.505430    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4" event={"ID":"d3bfb222-a8f8-47bc-a3cd-a62716211f0e","Type":"ContainerStarted","Data":"1613f07a0c36d13a3112b609f40ea87687cbae01303be2b29fd584d702785340"}
Oct 30 01:11:05.505540 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:05.505452    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4"
Oct 30 01:11:05.546444 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:05.546399    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p" podStartSLOduration=2.546386245 podStartE2EDuration="2.546386245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:05.542774029 +0000 UTC m=+2159.399681848" watchObservedRunningTime="2026-10-30 01:11:05.546386245 +0000 UTC m=+2159.403294000"
Oct 30 01:11:05.586114 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:05.586073    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4" podStartSLOduration=2.586059996 podStartE2EDuration="2.586059996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:05.581731211 +0000 UTC m=+2159.438638968" watchObservedRunningTime="2026-10-30 01:11:05.586059996 +0000 UTC m=+2159.442967751"
Oct 30 01:11:07.860039 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:07.860005    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:11:07.872777 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:07.872752    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:11:08.543818 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:08.543787    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:11:10.689274 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:10.689237    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:10.701557 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:10.701531    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:11.569914 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:11.569884    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:11:16.516901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:16.516870    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4"
Oct 30 01:11:16.517382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:16.516957    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p"
Oct 30 01:11:16.638735 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:11:16.638681    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:11:21.145844 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:21.145742    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:11:24.248731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:24.248247    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl" podUID="657a9d22-491f-4ef1-909b-a27511860627" containerName="main" probeResult="failure" output="Get \"https://10.134.1.0:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:11:24.363729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:24.362891    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 01:11:24.458252 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:24.457790    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:11:24.569999 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:24.569958    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk" podUID="5de95229-e8f1-483d-8540-15e1d33182ea" containerName="main" probeResult="failure" output="Get \"https://10.134.1.6:8000/ready\": context deadline exceeded"
Oct 30 01:11:25.594785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:25.594746    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf"]
Oct 30 01:11:25.601795 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:25.601769    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf"
Oct 30 01:11:25.618005 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:25.617975    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf"]
Oct 30 01:11:25.635518 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:25.635490    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:11:25.643720 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:25.643680    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:11:25.655820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:25.655789    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c36496d-2048-4e23-9e47-63c3fcb78bbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1161126e-lmjxf\" (UID: \"6c36496d-2048-4e23-9e47-63c3fcb78bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf"
Oct 30 01:11:25.655969 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:25.655953    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c36496d-2048-4e23-9e47-63c3fcb78bbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1161126e-lmjxf\" (UID: \"6c36496d-2048-4e23-9e47-63c3fcb78bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf"
Oct 30 01:11:25.656139 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:25.656121    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g7mv\" (UniqueName: \"kubernetes.io/projected/6c36496d-2048-4e23-9e47-63c3fcb78bbf-kube-api-access-8g7mv\") pod \"maas-api-revoke-sub-1161126e-lmjxf\" (UID: \"6c36496d-2048-4e23-9e47-63c3fcb78bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf"
Oct 30 01:11:25.758461 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:25.758416    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c36496d-2048-4e23-9e47-63c3fcb78bbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1161126e-lmjxf\" (UID: \"6c36496d-2048-4e23-9e47-63c3fcb78bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf"
Oct 30 01:11:25.758929 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:25.758892    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g7mv\" (UniqueName: \"kubernetes.io/projected/6c36496d-2048-4e23-9e47-63c3fcb78bbf-kube-api-access-8g7mv\") pod \"maas-api-revoke-sub-1161126e-lmjxf\" (UID: \"6c36496d-2048-4e23-9e47-63c3fcb78bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf"
Oct 30 01:11:25.759180 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:25.759155    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c36496d-2048-4e23-9e47-63c3fcb78bbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1161126e-lmjxf\" (UID: \"6c36496d-2048-4e23-9e47-63c3fcb78bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf"
Oct 30 01:11:25.759491 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:25.759464    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c36496d-2048-4e23-9e47-63c3fcb78bbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1161126e-lmjxf\" (UID: \"6c36496d-2048-4e23-9e47-63c3fcb78bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf"
Oct 30 01:11:25.761548 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:25.761525    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c36496d-2048-4e23-9e47-63c3fcb78bbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1161126e-lmjxf\" (UID: \"6c36496d-2048-4e23-9e47-63c3fcb78bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf"
Oct 30 01:11:25.794342 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:25.794314    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g7mv\" (UniqueName: \"kubernetes.io/projected/6c36496d-2048-4e23-9e47-63c3fcb78bbf-kube-api-access-8g7mv\") pod \"maas-api-revoke-sub-1161126e-lmjxf\" (UID: \"6c36496d-2048-4e23-9e47-63c3fcb78bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf"
Oct 30 01:11:25.920943 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:25.920854    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf"
Oct 30 01:11:26.293757 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:26.293696    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf"]
Oct 30 01:11:26.829361 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:26.829326    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf" event={"ID":"6c36496d-2048-4e23-9e47-63c3fcb78bbf","Type":"ContainerStarted","Data":"b29849b7249a3bf65d30024f5e2b61fb7ddba9c5eb6443ca5e2fa1394e2d6154"}
Oct 30 01:11:26.829818 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:26.829385    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf" event={"ID":"6c36496d-2048-4e23-9e47-63c3fcb78bbf","Type":"ContainerStarted","Data":"6ea042468eac548e973d6e28c3a9599075a91c9a6748ee8881a8ca3710885b20"}
Oct 30 01:11:26.893135 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:26.893075    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf" podStartSLOduration=1.893060983 podStartE2EDuration="1.893060983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:26.883175119 +0000 UTC m=+2180.740082875" watchObservedRunningTime="2026-10-30 01:11:26.893060983 +0000 UTC m=+2180.749968737"
Oct 30 01:11:27.839298 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:27.839265    6099 generic.go:356] "Generic (PLEG): container finished" podID="6c36496d-2048-4e23-9e47-63c3fcb78bbf" containerID="b29849b7249a3bf65d30024f5e2b61fb7ddba9c5eb6443ca5e2fa1394e2d6154" exitCode=0
Oct 30 01:11:27.839731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:27.839303    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf" event={"ID":"6c36496d-2048-4e23-9e47-63c3fcb78bbf","Type":"ContainerDied","Data":"b29849b7249a3bf65d30024f5e2b61fb7ddba9c5eb6443ca5e2fa1394e2d6154"}
Oct 30 01:11:29.097237 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:29.097205    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf"
Oct 30 01:11:29.243405 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:29.243303    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c36496d-2048-4e23-9e47-63c3fcb78bbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c36496d-2048-4e23-9e47-63c3fcb78bbf-maas-api-xks-ca\") pod \"6c36496d-2048-4e23-9e47-63c3fcb78bbf\" (UID: \"6c36496d-2048-4e23-9e47-63c3fcb78bbf\") "
Oct 30 01:11:29.243405 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:29.243381    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c36496d-2048-4e23-9e47-63c3fcb78bbf-kube-api-access-8g7mv\" (UniqueName: \"kubernetes.io/projected/6c36496d-2048-4e23-9e47-63c3fcb78bbf-kube-api-access-8g7mv\") pod \"6c36496d-2048-4e23-9e47-63c3fcb78bbf\" (UID: \"6c36496d-2048-4e23-9e47-63c3fcb78bbf\") "
Oct 30 01:11:29.243832 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:29.243810    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c36496d-2048-4e23-9e47-63c3fcb78bbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c36496d-2048-4e23-9e47-63c3fcb78bbf-maas-api-openshift-ca\") pod \"6c36496d-2048-4e23-9e47-63c3fcb78bbf\" (UID: \"6c36496d-2048-4e23-9e47-63c3fcb78bbf\") "
Oct 30 01:11:29.244341 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:29.244226    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c36496d-2048-4e23-9e47-63c3fcb78bbf-maas-api-openshift-ca" pod "6c36496d-2048-4e23-9e47-63c3fcb78bbf" (UID: "6c36496d-2048-4e23-9e47-63c3fcb78bbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:29.244636 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:29.244615    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c36496d-2048-4e23-9e47-63c3fcb78bbf-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:11:29.245669 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:29.245645    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c36496d-2048-4e23-9e47-63c3fcb78bbf-maas-api-xks-ca" pod "6c36496d-2048-4e23-9e47-63c3fcb78bbf" (UID: "6c36496d-2048-4e23-9e47-63c3fcb78bbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:29.245944 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:29.245921    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c36496d-2048-4e23-9e47-63c3fcb78bbf-kube-api-access-8g7mv" pod "6c36496d-2048-4e23-9e47-63c3fcb78bbf" (UID: "6c36496d-2048-4e23-9e47-63c3fcb78bbf"). InnerVolumeSpecName "kube-api-access-8g7mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:29.346670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:29.346631    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c36496d-2048-4e23-9e47-63c3fcb78bbf-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:11:29.346670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:29.346664    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g7mv\" (UniqueName: \"kubernetes.io/projected/6c36496d-2048-4e23-9e47-63c3fcb78bbf-kube-api-access-8g7mv\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:11:29.861843 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:29.861815    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf"
Oct 30 01:11:29.862025 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:29.861812    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf" event={"ID":"6c36496d-2048-4e23-9e47-63c3fcb78bbf","Type":"ContainerDied","Data":"6ea042468eac548e973d6e28c3a9599075a91c9a6748ee8881a8ca3710885b20"}
Oct 30 01:11:29.862025 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:29.861982    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ea042468eac548e973d6e28c3a9599075a91c9a6748ee8881a8ca3710885b20"
Oct 30 01:11:35.020672 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.020638    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"]
Oct 30 01:11:35.021673 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.021648    6099 cpu_manager.go:414] "containerMap: removing container" podUID="6c36496d-2048-4e23-9e47-63c3fcb78bbf" containerName="revoke-keys"
Oct 30 01:11:35.021807 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.021673    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c36496d-2048-4e23-9e47-63c3fcb78bbf" containerName="revoke-keys"
Oct 30 01:11:35.021908 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.021893    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c36496d-2048-4e23-9e47-63c3fcb78bbf" containerName="revoke-keys"
Oct 30 01:11:35.025071 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.025049    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.060160 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.060129    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"]
Oct 30 01:11:35.092059 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.092016    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-8e24a5-openshift-default-dockercfg-r6rgl\""
Oct 30 01:11:35.145727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.144991    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-workload-socket\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.145727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.145074    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-data\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.145727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.145140    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istiod-ca-cert\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.145727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.145226    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-token\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.145727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.145257    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-workload-certs\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.145727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.145286    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-credential-socket\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.145727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.145436    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-podinfo\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.145727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.145485    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7pvh\" (UniqueName: \"kubernetes.io/projected/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-kube-api-access-m7pvh\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.145727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.145527    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-envoy\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.248782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.248269    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-workload-socket\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.248782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.248343    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-data\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.248782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.248415    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istiod-ca-cert\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.248782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.248505    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-token\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.248782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.248542    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-workload-certs\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.248782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.248571    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-credential-socket\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.248782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.248756    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-podinfo\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.249352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.248813    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7pvh\" (UniqueName: \"kubernetes.io/projected/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-kube-api-access-m7pvh\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.249352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.248861    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-envoy\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.250205 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.250175    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-credential-socket\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.250467 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.250442    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-workload-certs\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.261143 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.256619    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-workload-socket\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.261143 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.257337    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istiod-ca-cert\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.266469 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.262741    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-envoy\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.267751 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.267376    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-podinfo\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.271135 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.270953    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-data\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.274700 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.274674    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7pvh\" (UniqueName: \"kubernetes.io/projected/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-kube-api-access-m7pvh\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.294764 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.294737    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-token\") pod \"e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") " pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.340893 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.340859    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:35.706887 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:11:35.706856    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eeb488e_9da7_4e8d_bfaf_0e8a4aa4d8ca.slice/crio-72567505f05713a6752dd09b6d1e788c03bc71a80b37aa5a81e524a456cfe776 WatchSource:0}: Error finding container 72567505f05713a6752dd09b6d1e788c03bc71a80b37aa5a81e524a456cfe776: Status 404 returned error can't find the container with id 72567505f05713a6752dd09b6d1e788c03bc71a80b37aa5a81e524a456cfe776
Oct 30 01:11:35.708780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.708619    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"]
Oct 30 01:11:35.710240 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.710197    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:11:35.710344 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.710285    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:11:35.710344 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.710329    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:11:35.722806 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.722779    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"]
Oct 30 01:11:35.728566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.728544    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"
Oct 30 01:11:35.746395 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.743646    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-320045-serving-cert\""
Oct 30 01:11:35.747895 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.747817    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"]
Oct 30 01:11:35.800748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.796868    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b07e253-0493-423b-a506-eaebd9952e34-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w\" (UID: \"5b07e253-0493-423b-a506-eaebd9952e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"
Oct 30 01:11:35.800748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.796913    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp94b\" (UniqueName: \"kubernetes.io/projected/5b07e253-0493-423b-a506-eaebd9952e34-kube-api-access-lp94b\") pod \"maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w\" (UID: \"5b07e253-0493-423b-a506-eaebd9952e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"
Oct 30 01:11:35.800748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.797135    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b07e253-0493-423b-a506-eaebd9952e34-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w\" (UID: \"5b07e253-0493-423b-a506-eaebd9952e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"
Oct 30 01:11:35.800748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.797182    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b07e253-0493-423b-a506-eaebd9952e34-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w\" (UID: \"5b07e253-0493-423b-a506-eaebd9952e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"
Oct 30 01:11:35.898688 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.898655    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b07e253-0493-423b-a506-eaebd9952e34-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w\" (UID: \"5b07e253-0493-423b-a506-eaebd9952e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"
Oct 30 01:11:35.898927 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.898697    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp94b\" (UniqueName: \"kubernetes.io/projected/5b07e253-0493-423b-a506-eaebd9952e34-kube-api-access-lp94b\") pod \"maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w\" (UID: \"5b07e253-0493-423b-a506-eaebd9952e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"
Oct 30 01:11:35.898927 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.898901    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b07e253-0493-423b-a506-eaebd9952e34-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w\" (UID: \"5b07e253-0493-423b-a506-eaebd9952e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"
Oct 30 01:11:35.899054 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.898931    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b07e253-0493-423b-a506-eaebd9952e34-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w\" (UID: \"5b07e253-0493-423b-a506-eaebd9952e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"
Oct 30 01:11:35.900256 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.900229    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b07e253-0493-423b-a506-eaebd9952e34-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w\" (UID: \"5b07e253-0493-423b-a506-eaebd9952e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"
Oct 30 01:11:35.902192 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.902165    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b07e253-0493-423b-a506-eaebd9952e34-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w\" (UID: \"5b07e253-0493-423b-a506-eaebd9952e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"
Oct 30 01:11:35.902508 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.902484    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b07e253-0493-423b-a506-eaebd9952e34-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w\" (UID: \"5b07e253-0493-423b-a506-eaebd9952e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"
Oct 30 01:11:35.921388 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.921355    6099 generic.go:356] "Generic (PLEG): container finished" podID="673e2d41-cb67-4956-bb96-959aa37fb924" containerID="95bd9bbccc2e6311c11db1dc7dcb68c104620dfa0d7824adb9c15f44f4c78976" exitCode=0
Oct 30 01:11:35.921661 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.921642    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb" event={"ID":"673e2d41-cb67-4956-bb96-959aa37fb924","Type":"ContainerDied","Data":"95bd9bbccc2e6311c11db1dc7dcb68c104620dfa0d7824adb9c15f44f4c78976"}
Oct 30 01:11:35.922175 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.922154    6099 scope.go:122] "RemoveContainer" containerID="95bd9bbccc2e6311c11db1dc7dcb68c104620dfa0d7824adb9c15f44f4c78976"
Oct 30 01:11:35.924991 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.924454    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:11:35.924991 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.924547    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:11:35.924991 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.924596    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:11:35.927121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.927088    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2" event={"ID":"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca","Type":"ContainerStarted","Data":"4847907fbfb9326367fc14657c0215056ea37a52cbdc03d808cddfb31db65231"}
Oct 30 01:11:35.927207 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.927141    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2" event={"ID":"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca","Type":"ContainerStarted","Data":"72567505f05713a6752dd09b6d1e788c03bc71a80b37aa5a81e524a456cfe776"}
Oct 30 01:11:35.934221 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:35.934201    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp94b\" (UniqueName: \"kubernetes.io/projected/5b07e253-0493-423b-a506-eaebd9952e34-kube-api-access-lp94b\") pod \"maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w\" (UID: \"5b07e253-0493-423b-a506-eaebd9952e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"
Oct 30 01:11:36.074843 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.074808    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"
Oct 30 01:11:36.085966 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.082339    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh"]
Oct 30 01:11:36.120799 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.119471    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh"
Oct 30 01:11:36.124487 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.124443    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c5e63e-5254-4961-94b2-4098db32e569-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e065abe8-hxnmh\" (UID: \"27c5e63e-5254-4961-94b2-4098db32e569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh"
Oct 30 01:11:36.124633 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.124575    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvxpg\" (UniqueName: \"kubernetes.io/projected/27c5e63e-5254-4961-94b2-4098db32e569-kube-api-access-hvxpg\") pod \"maas-api-revoke-sub-e065abe8-hxnmh\" (UID: \"27c5e63e-5254-4961-94b2-4098db32e569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh"
Oct 30 01:11:36.124633 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.124608    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c5e63e-5254-4961-94b2-4098db32e569-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e065abe8-hxnmh\" (UID: \"27c5e63e-5254-4961-94b2-4098db32e569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh"
Oct 30 01:11:36.137088 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.137035    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2" podStartSLOduration=2.137018095 podStartE2EDuration="2.137018095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:36.134964631 +0000 UTC m=+2189.991872387" watchObservedRunningTime="2026-10-30 01:11:36.137018095 +0000 UTC m=+2189.993925850"
Oct 30 01:11:36.144396 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.137808    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:11:36.144567 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.137882    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:11:36.171009 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.170425    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh"]
Oct 30 01:11:36.371532 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.363602    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c5e63e-5254-4961-94b2-4098db32e569-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e065abe8-hxnmh\" (UID: \"27c5e63e-5254-4961-94b2-4098db32e569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh"
Oct 30 01:11:36.373056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.371963    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvxpg\" (UniqueName: \"kubernetes.io/projected/27c5e63e-5254-4961-94b2-4098db32e569-kube-api-access-hvxpg\") pod \"maas-api-revoke-sub-e065abe8-hxnmh\" (UID: \"27c5e63e-5254-4961-94b2-4098db32e569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh"
Oct 30 01:11:36.373056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.372023    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c5e63e-5254-4961-94b2-4098db32e569-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e065abe8-hxnmh\" (UID: \"27c5e63e-5254-4961-94b2-4098db32e569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh"
Oct 30 01:11:36.375721 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.375682    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c5e63e-5254-4961-94b2-4098db32e569-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e065abe8-hxnmh\" (UID: \"27c5e63e-5254-4961-94b2-4098db32e569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh"
Oct 30 01:11:36.380722 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.378237    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body=
Oct 30 01:11:36.380722 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.378293    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2" podUID="2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused"
Oct 30 01:11:36.391356 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.391330    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c5e63e-5254-4961-94b2-4098db32e569-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e065abe8-hxnmh\" (UID: \"27c5e63e-5254-4961-94b2-4098db32e569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh"
Oct 30 01:11:36.412759 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.411265    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvxpg\" (UniqueName: \"kubernetes.io/projected/27c5e63e-5254-4961-94b2-4098db32e569-kube-api-access-hvxpg\") pod \"maas-api-revoke-sub-e065abe8-hxnmh\" (UID: \"27c5e63e-5254-4961-94b2-4098db32e569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh"
Oct 30 01:11:36.466377 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.466347    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:36.521575 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.521544    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh"
Oct 30 01:11:36.545337 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.545301    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"]
Oct 30 01:11:36.546558 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:11:36.546527    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b07e253_0493_423b_a506_eaebd9952e34.slice/crio-4201ed584055338bbc72b72bec7c89b4988f82ab35d9c40f532a1e21a7d068d3 WatchSource:0}: Error finding container 4201ed584055338bbc72b72bec7c89b4988f82ab35d9c40f532a1e21a7d068d3: Status 404 returned error can't find the container with id 4201ed584055338bbc72b72bec7c89b4988f82ab35d9c40f532a1e21a7d068d3
Oct 30 01:11:36.861879 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:11:36.861842    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c5e63e_5254_4961_94b2_4098db32e569.slice/crio-6028fa3549428d72b14b2e8694861b8d9da744caee089ffa5f01057412d65e35 WatchSource:0}: Error finding container 6028fa3549428d72b14b2e8694861b8d9da744caee089ffa5f01057412d65e35: Status 404 returned error can't find the container with id 6028fa3549428d72b14b2e8694861b8d9da744caee089ffa5f01057412d65e35
Oct 30 01:11:36.863156 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.863127    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh"]
Oct 30 01:11:36.943132 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.943107    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb" event={"ID":"673e2d41-cb67-4956-bb96-959aa37fb924","Type":"ContainerStarted","Data":"45c62a6b60da5713aa75144e46981591c3b1806df2684d95cbeb42d1711b868f"}
Oct 30 01:11:36.944786 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.944758    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w" event={"ID":"5b07e253-0493-423b-a506-eaebd9952e34","Type":"ContainerStarted","Data":"4201ed584055338bbc72b72bec7c89b4988f82ab35d9c40f532a1e21a7d068d3"}
Oct 30 01:11:36.946417 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:36.946387    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh" event={"ID":"27c5e63e-5254-4961-94b2-4098db32e569","Type":"ContainerStarted","Data":"6028fa3549428d72b14b2e8694861b8d9da744caee089ffa5f01057412d65e35"}
Oct 30 01:11:37.349471 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:37.348988    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:37.818677 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:37.818641    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:11:37.820322 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:37.820294    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 30 01:11:37.820458 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:37.820342    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb" podUID="673e2d41-cb67-4956-bb96-959aa37fb924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 30 01:11:37.956780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:37.956700    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh" event={"ID":"27c5e63e-5254-4961-94b2-4098db32e569","Type":"ContainerStarted","Data":"08e21d7d3c5cbf726abbb3f3ad2d538725a2994c45a3abaf23016330b905ca8a"}
Oct 30 01:11:37.958500 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:37.958471    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w" event={"ID":"5b07e253-0493-423b-a506-eaebd9952e34","Type":"ContainerStarted","Data":"e8a2d8dcad1c5723e706475af6eafee87f8d680979c575f13b0976abadb872f0"}
Oct 30 01:11:37.958678 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:37.958517    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"
Oct 30 01:11:37.958941 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:37.958920    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:37.959797 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:37.959777    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:11:38.015113 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:38.015053    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh" podStartSLOduration=2.015035187 podStartE2EDuration="2.015035187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:38.014102487 +0000 UTC m=+2191.871010243" watchObservedRunningTime="2026-10-30 01:11:38.015035187 +0000 UTC m=+2191.871942943"
Oct 30 01:11:38.058700 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:38.058649    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w" podStartSLOduration=2.700633598 podStartE2EDuration="3.058635511s" totalImagesPullingTime="358.001913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:38.055002254 +0000 UTC m=+2191.911910010" watchObservedRunningTime="2026-10-30 01:11:38.058635511 +0000 UTC m=+2191.915543323"
Oct 30 01:11:38.818460 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:38.818429    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 30 01:11:38.819001 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:38.818478    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb" podUID="673e2d41-cb67-4956-bb96-959aa37fb924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 30 01:11:38.993580 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:38.993546    6099 generic.go:356] "Generic (PLEG): container finished" podID="27c5e63e-5254-4961-94b2-4098db32e569" containerID="08e21d7d3c5cbf726abbb3f3ad2d538725a2994c45a3abaf23016330b905ca8a" exitCode=0
Oct 30 01:11:38.993801 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:38.993626    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh" event={"ID":"27c5e63e-5254-4961-94b2-4098db32e569","Type":"ContainerDied","Data":"08e21d7d3c5cbf726abbb3f3ad2d538725a2994c45a3abaf23016330b905ca8a"}
Oct 30 01:11:39.460502 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:39.460466    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:11:39.818873 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:39.818838    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 30 01:11:39.819269 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:39.818896    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb" podUID="673e2d41-cb67-4956-bb96-959aa37fb924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 30 01:11:40.222679 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.222649    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh"
Oct 30 01:11:40.408738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.408641    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27c5e63e-5254-4961-94b2-4098db32e569-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c5e63e-5254-4961-94b2-4098db32e569-maas-api-xks-ca\") pod \"27c5e63e-5254-4961-94b2-4098db32e569\" (UID: \"27c5e63e-5254-4961-94b2-4098db32e569\") "
Oct 30 01:11:40.409032 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.408992    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c5e63e-5254-4961-94b2-4098db32e569-kube-api-access-hvxpg\" (UniqueName: \"kubernetes.io/projected/27c5e63e-5254-4961-94b2-4098db32e569-kube-api-access-hvxpg\") pod \"27c5e63e-5254-4961-94b2-4098db32e569\" (UID: \"27c5e63e-5254-4961-94b2-4098db32e569\") "
Oct 30 01:11:40.409214 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.409198    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27c5e63e-5254-4961-94b2-4098db32e569-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c5e63e-5254-4961-94b2-4098db32e569-maas-api-openshift-ca\") pod \"27c5e63e-5254-4961-94b2-4098db32e569\" (UID: \"27c5e63e-5254-4961-94b2-4098db32e569\") "
Oct 30 01:11:40.409629 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.409596    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c5e63e-5254-4961-94b2-4098db32e569-maas-api-openshift-ca" pod "27c5e63e-5254-4961-94b2-4098db32e569" (UID: "27c5e63e-5254-4961-94b2-4098db32e569"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:40.410078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.410055    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c5e63e-5254-4961-94b2-4098db32e569-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:11:40.411323 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.411294    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c5e63e-5254-4961-94b2-4098db32e569-kube-api-access-hvxpg" pod "27c5e63e-5254-4961-94b2-4098db32e569" (UID: "27c5e63e-5254-4961-94b2-4098db32e569"). InnerVolumeSpecName "kube-api-access-hvxpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:40.412051 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.412026    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c5e63e-5254-4961-94b2-4098db32e569-maas-api-xks-ca" pod "27c5e63e-5254-4961-94b2-4098db32e569" (UID: "27c5e63e-5254-4961-94b2-4098db32e569"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:40.511390 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.511355    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c5e63e-5254-4961-94b2-4098db32e569-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:11:40.511390 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.511389    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvxpg\" (UniqueName: \"kubernetes.io/projected/27c5e63e-5254-4961-94b2-4098db32e569-kube-api-access-hvxpg\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:11:40.521852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.521820    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42"]
Oct 30 01:11:40.522815 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.522783    6099 cpu_manager.go:414] "containerMap: removing container" podUID="27c5e63e-5254-4961-94b2-4098db32e569" containerName="revoke-keys"
Oct 30 01:11:40.522815 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.522816    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c5e63e-5254-4961-94b2-4098db32e569" containerName="revoke-keys"
Oct 30 01:11:40.522983 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.522969    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c5e63e-5254-4961-94b2-4098db32e569" containerName="revoke-keys"
Oct 30 01:11:40.525804 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.525785    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42"
Oct 30 01:11:40.547600 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.547575    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42"]
Oct 30 01:11:40.549918 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.549899    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-8e24a5\""
Oct 30 01:11:40.551686 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.551667    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-8e24a5-dockercfg-7lbg2\""
Oct 30 01:11:40.749784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.732056    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7dk4\" (UniqueName: \"kubernetes.io/projected/53ccfecd-0ce5-4382-a5e6-e8acd989e4e1-kube-api-access-k7dk4\") pod \"payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42\" (UID: \"53ccfecd-0ce5-4382-a5e6-e8acd989e4e1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42"
Oct 30 01:11:40.749784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.732136    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ccfecd-0ce5-4382-a5e6-e8acd989e4e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42\" (UID: \"53ccfecd-0ce5-4382-a5e6-e8acd989e4e1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42"
Oct 30 01:11:40.832731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.823799    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 30 01:11:40.832731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.823863    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb" podUID="673e2d41-cb67-4956-bb96-959aa37fb924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 30 01:11:40.867729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.867070    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7dk4\" (UniqueName: \"kubernetes.io/projected/53ccfecd-0ce5-4382-a5e6-e8acd989e4e1-kube-api-access-k7dk4\") pod \"payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42\" (UID: \"53ccfecd-0ce5-4382-a5e6-e8acd989e4e1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42"
Oct 30 01:11:40.867729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.867142    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ccfecd-0ce5-4382-a5e6-e8acd989e4e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42\" (UID: \"53ccfecd-0ce5-4382-a5e6-e8acd989e4e1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42"
Oct 30 01:11:40.873153 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.868553    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ccfecd-0ce5-4382-a5e6-e8acd989e4e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42\" (UID: \"53ccfecd-0ce5-4382-a5e6-e8acd989e4e1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42"
Oct 30 01:11:40.936734 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:40.931059    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7dk4\" (UniqueName: \"kubernetes.io/projected/53ccfecd-0ce5-4382-a5e6-e8acd989e4e1-kube-api-access-k7dk4\") pod \"payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42\" (UID: \"53ccfecd-0ce5-4382-a5e6-e8acd989e4e1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42"
Oct 30 01:11:41.015005 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:41.014913    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh"
Oct 30 01:11:41.015182 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:41.014910    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh" event={"ID":"27c5e63e-5254-4961-94b2-4098db32e569","Type":"ContainerDied","Data":"6028fa3549428d72b14b2e8694861b8d9da744caee089ffa5f01057412d65e35"}
Oct 30 01:11:41.015182 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:41.015065    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6028fa3549428d72b14b2e8694861b8d9da744caee089ffa5f01057412d65e35"
Oct 30 01:11:41.143343 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:41.143316    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42"
Oct 30 01:11:41.498213 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:41.498180    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42"]
Oct 30 01:11:41.818398 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:41.818366    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 30 01:11:41.818587 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:41.818415    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb" podUID="673e2d41-cb67-4956-bb96-959aa37fb924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 30 01:11:42.024250 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:42.024214    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42" event={"ID":"53ccfecd-0ce5-4382-a5e6-e8acd989e4e1","Type":"ContainerStarted","Data":"f5720116286575ea8d5b331f78ac5238b62d9dd5996bddacfd7567413292bdb5"}
Oct 30 01:11:42.024658 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:42.024282    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42"
Oct 30 01:11:42.024658 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:42.024307    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42" event={"ID":"53ccfecd-0ce5-4382-a5e6-e8acd989e4e1","Type":"ContainerStarted","Data":"1aad9c3371e8c4c67d01a1e4c35701477ef5d0f79729f70f2622a6a4481a5bea"}
Oct 30 01:11:42.063346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:42.063294    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42" podStartSLOduration=2.0632794 podStartE2EDuration="2.0632794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:42.062450279 +0000 UTC m=+2195.919358033" watchObservedRunningTime="2026-10-30 01:11:42.0632794 +0000 UTC m=+2195.920187156"
Oct 30 01:11:42.823700 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:42.823665    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:11:42.824925 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:42.824902    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:11:43.999544 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:43.999516    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"
Oct 30 01:11:46.631560 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:11:46.631523    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:11:53.045079 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:11:53.045046    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42"
Oct 30 01:12:05.195144 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:05.195108    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"]
Oct 30 01:12:14.924410 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:14.922576    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"]
Oct 30 01:12:14.928800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:14.928775    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"]
Oct 30 01:12:14.928954 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:14.928868    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:12:14.945580 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:14.945552    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-8e24a5-serving-cert\""
Oct 30 01:12:14.993057 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:14.983061    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/330745dd-20f6-4e76-8513-3abf69325b80-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz\" (UID: \"330745dd-20f6-4e76-8513-3abf69325b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:12:14.994582 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:14.994466    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/330745dd-20f6-4e76-8513-3abf69325b80-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz\" (UID: \"330745dd-20f6-4e76-8513-3abf69325b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:12:14.995050 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:14.994924    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ddns\" (UniqueName: \"kubernetes.io/projected/330745dd-20f6-4e76-8513-3abf69325b80-kube-api-access-9ddns\") pod \"maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz\" (UID: \"330745dd-20f6-4e76-8513-3abf69325b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:12:14.995481 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:14.995369    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/330745dd-20f6-4e76-8513-3abf69325b80-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz\" (UID: \"330745dd-20f6-4e76-8513-3abf69325b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:12:15.097299 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:15.097260    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/330745dd-20f6-4e76-8513-3abf69325b80-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz\" (UID: \"330745dd-20f6-4e76-8513-3abf69325b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:12:15.097468 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:15.097332    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ddns\" (UniqueName: \"kubernetes.io/projected/330745dd-20f6-4e76-8513-3abf69325b80-kube-api-access-9ddns\") pod \"maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz\" (UID: \"330745dd-20f6-4e76-8513-3abf69325b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:12:15.097468 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:15.097392    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/330745dd-20f6-4e76-8513-3abf69325b80-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz\" (UID: \"330745dd-20f6-4e76-8513-3abf69325b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:12:15.097546 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:15.097465    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/330745dd-20f6-4e76-8513-3abf69325b80-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz\" (UID: \"330745dd-20f6-4e76-8513-3abf69325b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:12:15.098260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:15.098240    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/330745dd-20f6-4e76-8513-3abf69325b80-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz\" (UID: \"330745dd-20f6-4e76-8513-3abf69325b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:12:15.098743 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:12:15.098696    6099 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-serving-cert: secret "maas-api-e2e-shared-b-w5-8e24a5-serving-cert" not found
Oct 30 01:12:15.098858 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:12:15.098803    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/330745dd-20f6-4e76-8513-3abf69325b80-maas-api-tls podName:330745dd-20f6-4e76-8513-3abf69325b80 nodeName:}" failed. No retries permitted until 2026-10-30 01:12:15.598782855 +0000 UTC m=+2229.455690609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/330745dd-20f6-4e76-8513-3abf69325b80-maas-api-tls") pod "maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz" (UID: "330745dd-20f6-4e76-8513-3abf69325b80") : secret "maas-api-e2e-shared-b-w5-8e24a5-serving-cert" not found
Oct 30 01:12:15.100791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:15.100750    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/330745dd-20f6-4e76-8513-3abf69325b80-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz\" (UID: \"330745dd-20f6-4e76-8513-3abf69325b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:12:15.153267 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:15.153184    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ddns\" (UniqueName: \"kubernetes.io/projected/330745dd-20f6-4e76-8513-3abf69325b80-kube-api-access-9ddns\") pod \"maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz\" (UID: \"330745dd-20f6-4e76-8513-3abf69325b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:12:15.606789 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:15.606746    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/330745dd-20f6-4e76-8513-3abf69325b80-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz\" (UID: \"330745dd-20f6-4e76-8513-3abf69325b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:12:15.609207 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:15.609178    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/330745dd-20f6-4e76-8513-3abf69325b80-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz\" (UID: \"330745dd-20f6-4e76-8513-3abf69325b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:12:15.847819 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:15.847789    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:12:16.196265 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:12:16.196236    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330745dd_20f6_4e76_8513_3abf69325b80.slice/crio-e9ee3731a13291c3aafabaeb926cb02f74a939d67672995b9e354010946e6f76 WatchSource:0}: Error finding container e9ee3731a13291c3aafabaeb926cb02f74a939d67672995b9e354010946e6f76: Status 404 returned error can't find the container with id e9ee3731a13291c3aafabaeb926cb02f74a939d67672995b9e354010946e6f76
Oct 30 01:12:16.196915 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:16.196546    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"]
Oct 30 01:12:16.634744 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:12:16.634671    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:12:16.740331 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:16.740279    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz" event={"ID":"330745dd-20f6-4e76-8513-3abf69325b80","Type":"ContainerStarted","Data":"2cb8b78c1f0879cbd38771a98e3160409f70d38abcdb61f0c8ee8a890aeb2434"}
Oct 30 01:12:16.740523 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:16.740346    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz" event={"ID":"330745dd-20f6-4e76-8513-3abf69325b80","Type":"ContainerStarted","Data":"e9ee3731a13291c3aafabaeb926cb02f74a939d67672995b9e354010946e6f76"}
Oct 30 01:12:16.740523 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:16.740372    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:12:16.783702 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:16.783646    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz" podStartSLOduration=2.424790882 podStartE2EDuration="2.783632992s" totalImagesPullingTime="358.84211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:16.781719725 +0000 UTC m=+2230.638627481" watchObservedRunningTime="2026-10-30 01:12:16.783632992 +0000 UTC m=+2230.640540750"
Oct 30 01:12:19.030700 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.030645    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc"]
Oct 30 01:12:19.042186 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.042158    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc67105e-hhbcc"]
Oct 30 01:12:19.302373 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.302339    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"]
Oct 30 01:12:19.354644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.354605    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"]
Oct 30 01:12:19.354872 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.354739    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.369466 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.369431    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-f69104-openshift-default-dockercfg-2b6hn\""
Oct 30 01:12:19.506728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.500829    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-podinfo\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.507116 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.507079    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2c83697-c779-4359-804d-b6b4b4d5e77c-istiod-ca-cert\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.518984 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.518950    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-token\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.519277 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.519255    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-workload-socket\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.519605 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.519583    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfs4x\" (UniqueName: \"kubernetes.io/projected/e2c83697-c779-4359-804d-b6b4b4d5e77c-kube-api-access-mfs4x\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.519881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.519857    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-workload-certs\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.520079 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.520056    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-credential-socket\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.520204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.520188    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-envoy\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.520380 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.520362    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-data\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.634811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.622896    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-token\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.634811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.622969    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-workload-socket\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.634811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.623080    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfs4x\" (UniqueName: \"kubernetes.io/projected/e2c83697-c779-4359-804d-b6b4b4d5e77c-kube-api-access-mfs4x\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.634811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.623180    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-workload-certs\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.634811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.623209    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-credential-socket\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.634811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.623230    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-envoy\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.634811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.623275    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-data\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.634811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.623304    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-podinfo\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.634811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.623377    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2c83697-c779-4359-804d-b6b4b4d5e77c-istiod-ca-cert\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.634811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.624258    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2c83697-c779-4359-804d-b6b4b4d5e77c-istiod-ca-cert\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.634811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.624877    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-workload-socket\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.638966 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.638092    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-workload-certs\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.638966 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.638551    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-credential-socket\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.639964 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.639926    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-data\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.646961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.646895    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-envoy\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.649119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.648006    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-token\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.651635 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.651617    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-podinfo\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.651778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.651751    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfs4x\" (UniqueName: \"kubernetes.io/projected/e2c83697-c779-4359-804d-b6b4b4d5e77c-kube-api-access-mfs4x\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:19.764455 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:19.764418    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:20.192483 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:20.192453    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"]
Oct 30 01:12:20.193134 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:12:20.193095    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c83697_c779_4359_804d_b6b4b4d5e77c.slice/crio-ae336b57e4202ba0f18986d534ca3a7c2e72d4a27317add78bcf4dab4def721a WatchSource:0}: Error finding container ae336b57e4202ba0f18986d534ca3a7c2e72d4a27317add78bcf4dab4def721a: Status 404 returned error can't find the container with id ae336b57e4202ba0f18986d534ca3a7c2e72d4a27317add78bcf4dab4def721a
Oct 30 01:12:20.196752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:20.196674    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:12:20.196868 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:20.196776    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:12:20.196868 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:20.196813    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:12:20.236755 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:20.236689    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s" podUID="bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7" containerName="main" containerID="cri-o://35d4b24e7602bb0a5ee12a49033dbe19ad63d15782c597e486e9f2a6797dbcd8" gracePeriod=45
Oct 30 01:12:20.265153 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:20.265122    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83eafb1e-2ca8-4a11-bea0-04ed87a35dab" path="/var/lib/kubelet/pods/83eafb1e-2ca8-4a11-bea0-04ed87a35dab/volumes"
Oct 30 01:12:20.784153 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:20.784067    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz" event={"ID":"e2c83697-c779-4359-804d-b6b4b4d5e77c","Type":"ContainerStarted","Data":"1f43de5288b1add050adc94449f17f9bcfbad724425c6e912e546a11f767230c"}
Oct 30 01:12:20.784153 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:20.784121    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz" event={"ID":"e2c83697-c779-4359-804d-b6b4b4d5e77c","Type":"ContainerStarted","Data":"ae336b57e4202ba0f18986d534ca3a7c2e72d4a27317add78bcf4dab4def721a"}
Oct 30 01:12:20.785958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:20.785932    6099 generic.go:356] "Generic (PLEG): container finished" podID="bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7" containerID="35d4b24e7602bb0a5ee12a49033dbe19ad63d15782c597e486e9f2a6797dbcd8" exitCode=0
Oct 30 01:12:20.786095 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:20.786002    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s" event={"ID":"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7","Type":"ContainerDied","Data":"35d4b24e7602bb0a5ee12a49033dbe19ad63d15782c597e486e9f2a6797dbcd8"}
Oct 30 01:12:20.858518 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:20.858454    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz" podStartSLOduration=1.8584349310000001 podStartE2EDuration="1.858434931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:20.839283395 +0000 UTC m=+2234.696191150" watchObservedRunningTime="2026-10-30 01:12:20.858434931 +0000 UTC m=+2234.715342692"
Oct 30 01:12:20.965543 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:20.965148    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s" podUID="bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.241:8000/ready\": dial tcp 10.134.0.241:8000: connect: connection refused"
Oct 30 01:12:21.138629 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.138602    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:12:21.279788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.279751    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-home\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-home\") pod \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") "
Oct 30 01:12:21.280317 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.279817    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-dshm\") pod \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") "
Oct 30 01:12:21.280317 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.279919    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-tmp-dir\") pod \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") "
Oct 30 01:12:21.280317 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.279939    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-tls-certs\") pod \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") "
Oct 30 01:12:21.280317 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.279968    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-kube-api-access-5bbvp\" (UniqueName: \"kubernetes.io/projected/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-kube-api-access-5bbvp\") pod \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") "
Oct 30 01:12:21.280317 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.279996    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-home" pod "bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7" (UID: "bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:21.280317 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.280050    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-model-cache\") pod \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\" (UID: \"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7\") "
Oct 30 01:12:21.280317 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.280186    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-tmp-dir" pod "bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7" (UID: "bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:21.280656 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.280338    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-model-cache" pod "bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7" (UID: "bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:21.281027 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.281006    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:21.281075 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.281035    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:21.281075 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.281052    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:21.282201 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.282177    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-tls-certs" pod "bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7" (UID: "bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:21.282300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.282182    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-dshm" pod "bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7" (UID: "bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:21.282300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.282240    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-kube-api-access-5bbvp" pod "bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7" (UID: "bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7"). InnerVolumeSpecName "kube-api-access-5bbvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:21.382551 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.382482    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:21.382551 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.382508    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:21.382551 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.382518    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bbvp\" (UniqueName: \"kubernetes.io/projected/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7-kube-api-access-5bbvp\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:21.765443 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.765366    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:21.771390 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.771364    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:21.796063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.796035    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"
Oct 30 01:12:21.796188 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.796079    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s" event={"ID":"bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7","Type":"ContainerDied","Data":"820552c76fce71fe14f9832685a636a77b42c9a10d24d7b6e795a2e37049e2bf"}
Oct 30 01:12:21.796188 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.796122    6099 scope.go:122] "RemoveContainer" containerID="35d4b24e7602bb0a5ee12a49033dbe19ad63d15782c597e486e9f2a6797dbcd8"
Oct 30 01:12:21.796422 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.796400    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:21.797612 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.797591    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:12:21.918425 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.910222    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"]
Oct 30 01:12:21.918425 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:21.916643    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-16c1bb/auth-test-model-16c1bb-kserve-8685d9cb69-ww76s"]
Oct 30 01:12:22.264758 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:22.264687    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7" path="/var/lib/kubelet/pods/bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7/volumes"
Oct 30 01:12:22.324783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:22.324746    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"]
Oct 30 01:12:22.325331 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:22.325000    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w" podUID="cc360f77-72ad-4729-a4e7-bf542d69b142" containerName="istio-proxy" containerID="cri-o://3fd6fcdc43306ca714f166a92425d2fa4f40230e6bf49fe722d7600d0df5f4c4" gracePeriod=30
Oct 30 01:12:22.603154 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:22.603125    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"]
Oct 30 01:12:22.603399 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:22.603335    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p" podUID="19331f3a-39f8-46af-8b24-ccfa677abecf" containerName="istio-proxy" containerID="cri-o://244f27a398e1fa99891444d79abe5b6c0ab86d6a62e429fdadc21e37bebbb848" gracePeriod=30
Oct 30 01:12:22.758627 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:22.758599    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:12:24.593489 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:24.593447    6099 patch_prober.go:29] interesting pod/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:12:24.593938 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:24.593514    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w" podUID="cc360f77-72ad-4729-a4e7-bf542d69b142" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:12:27.871310 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:27.871246    6099 generic.go:356] "Generic (PLEG): container finished" podID="cc360f77-72ad-4729-a4e7-bf542d69b142" containerID="3fd6fcdc43306ca714f166a92425d2fa4f40230e6bf49fe722d7600d0df5f4c4" exitCode=0
Oct 30 01:12:27.871740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:27.871356    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w" event={"ID":"cc360f77-72ad-4729-a4e7-bf542d69b142","Type":"ContainerDied","Data":"3fd6fcdc43306ca714f166a92425d2fa4f40230e6bf49fe722d7600d0df5f4c4"}
Oct 30 01:12:27.871740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:27.871399    6099 scope.go:122] "RemoveContainer" containerID="9195d5f4afb0146b39bad6c325ade3192ec9f7933ed4cefb337912cbeb1d44e8"
Oct 30 01:12:27.879113 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:27.878936    6099 generic.go:356] "Generic (PLEG): container finished" podID="19331f3a-39f8-46af-8b24-ccfa677abecf" containerID="244f27a398e1fa99891444d79abe5b6c0ab86d6a62e429fdadc21e37bebbb848" exitCode=0
Oct 30 01:12:27.879113 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:27.878983    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p" event={"ID":"19331f3a-39f8-46af-8b24-ccfa677abecf","Type":"ContainerDied","Data":"244f27a398e1fa99891444d79abe5b6c0ab86d6a62e429fdadc21e37bebbb848"}
Oct 30 01:12:27.908899 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:27.908092    6099 scope.go:122] "RemoveContainer" containerID="151d441a1fa03d80f442bf6b6a3e8efb2b26e99b9fa183840e413776eb699860"
Oct 30 01:12:28.202569 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.202389    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 01:12:28.210550 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.210526    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 01:12:28.229320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.226992    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-data\") pod \"cc360f77-72ad-4729-a4e7-bf542d69b142\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") "
Oct 30 01:12:28.229320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.227053    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-workload-certs\") pod \"19331f3a-39f8-46af-8b24-ccfa677abecf\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") "
Oct 30 01:12:28.229320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.227080    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-data\") pod \"19331f3a-39f8-46af-8b24-ccfa677abecf\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") "
Oct 30 01:12:28.229320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.227145    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-token\" (UniqueName: \"kubernetes.io/projected/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-token\") pod \"cc360f77-72ad-4729-a4e7-bf542d69b142\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") "
Oct 30 01:12:28.229320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.227221    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc360f77-72ad-4729-a4e7-bf542d69b142-kube-api-access-4gznp\" (UniqueName: \"kubernetes.io/projected/cc360f77-72ad-4729-a4e7-bf542d69b142-kube-api-access-4gznp\") pod \"cc360f77-72ad-4729-a4e7-bf542d69b142\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") "
Oct 30 01:12:28.229320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.227298    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-data" pod "cc360f77-72ad-4729-a4e7-bf542d69b142" (UID: "cc360f77-72ad-4729-a4e7-bf542d69b142"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:28.229320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.227335    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-workload-socket\") pod \"19331f3a-39f8-46af-8b24-ccfa677abecf\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") "
Oct 30 01:12:28.229320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.227370    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-workload-certs" pod "19331f3a-39f8-46af-8b24-ccfa677abecf" (UID: "19331f3a-39f8-46af-8b24-ccfa677abecf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:28.229320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.227381    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-credential-socket\") pod \"cc360f77-72ad-4729-a4e7-bf542d69b142\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") "
Oct 30 01:12:28.229320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.227433    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19331f3a-39f8-46af-8b24-ccfa677abecf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19331f3a-39f8-46af-8b24-ccfa677abecf-istiod-ca-cert\") pod \"19331f3a-39f8-46af-8b24-ccfa677abecf\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") "
Oct 30 01:12:28.229320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.227518    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-podinfo\") pod \"cc360f77-72ad-4729-a4e7-bf542d69b142\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") "
Oct 30 01:12:28.229320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.227568    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19331f3a-39f8-46af-8b24-ccfa677abecf-kube-api-access-g9pp2\" (UniqueName: \"kubernetes.io/projected/19331f3a-39f8-46af-8b24-ccfa677abecf-kube-api-access-g9pp2\") pod \"19331f3a-39f8-46af-8b24-ccfa677abecf\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") "
Oct 30 01:12:28.229320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.227599    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-workload-socket" pod "19331f3a-39f8-46af-8b24-ccfa677abecf" (UID: "19331f3a-39f8-46af-8b24-ccfa677abecf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.231463    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-data" pod "19331f3a-39f8-46af-8b24-ccfa677abecf" (UID: "19331f3a-39f8-46af-8b24-ccfa677abecf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.231832    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-credential-socket" pod "cc360f77-72ad-4729-a4e7-bf542d69b142" (UID: "cc360f77-72ad-4729-a4e7-bf542d69b142"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.227687    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-token\" (UniqueName: \"kubernetes.io/projected/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-token\") pod \"19331f3a-39f8-46af-8b24-ccfa677abecf\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") "
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.232260    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19331f3a-39f8-46af-8b24-ccfa677abecf-istiod-ca-cert" pod "19331f3a-39f8-46af-8b24-ccfa677abecf" (UID: "19331f3a-39f8-46af-8b24-ccfa677abecf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.232297    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc360f77-72ad-4729-a4e7-bf542d69b142-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc360f77-72ad-4729-a4e7-bf542d69b142-istiod-ca-cert\") pod \"cc360f77-72ad-4729-a4e7-bf542d69b142\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") "
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.232359    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-envoy\") pod \"cc360f77-72ad-4729-a4e7-bf542d69b142\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") "
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.232434    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-envoy\") pod \"19331f3a-39f8-46af-8b24-ccfa677abecf\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") "
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.232468    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-credential-socket\") pod \"19331f3a-39f8-46af-8b24-ccfa677abecf\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") "
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.232516    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-podinfo\") pod \"19331f3a-39f8-46af-8b24-ccfa677abecf\" (UID: \"19331f3a-39f8-46af-8b24-ccfa677abecf\") "
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.232574    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-workload-certs\") pod \"cc360f77-72ad-4729-a4e7-bf542d69b142\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") "
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.232627    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-workload-socket\") pod \"cc360f77-72ad-4729-a4e7-bf542d69b142\" (UID: \"cc360f77-72ad-4729-a4e7-bf542d69b142\") "
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.233211    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-token" pod "cc360f77-72ad-4729-a4e7-bf542d69b142" (UID: "cc360f77-72ad-4729-a4e7-bf542d69b142"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.233799    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc360f77-72ad-4729-a4e7-bf542d69b142-kube-api-access-4gznp" pod "cc360f77-72ad-4729-a4e7-bf542d69b142" (UID: "cc360f77-72ad-4729-a4e7-bf542d69b142"). InnerVolumeSpecName "kube-api-access-4gznp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.234489    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-token" pod "19331f3a-39f8-46af-8b24-ccfa677abecf" (UID: "19331f3a-39f8-46af-8b24-ccfa677abecf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.234877    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.234902    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.234925    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19331f3a-39f8-46af-8b24-ccfa677abecf-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.234939    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.234953    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.234966    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.234982    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.234996    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.235009    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gznp\" (UniqueName: \"kubernetes.io/projected/cc360f77-72ad-4729-a4e7-bf542d69b142-kube-api-access-4gznp\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.235258    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-workload-certs" pod "cc360f77-72ad-4729-a4e7-bf542d69b142" (UID: "cc360f77-72ad-4729-a4e7-bf542d69b142"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.235418    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-podinfo" pod "cc360f77-72ad-4729-a4e7-bf542d69b142" (UID: "cc360f77-72ad-4729-a4e7-bf542d69b142"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.235433    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-workload-socket" pod "cc360f77-72ad-4729-a4e7-bf542d69b142" (UID: "cc360f77-72ad-4729-a4e7-bf542d69b142"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.235623    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-credential-socket" pod "19331f3a-39f8-46af-8b24-ccfa677abecf" (UID: "19331f3a-39f8-46af-8b24-ccfa677abecf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.235721    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc360f77-72ad-4729-a4e7-bf542d69b142-istiod-ca-cert" pod "cc360f77-72ad-4729-a4e7-bf542d69b142" (UID: "cc360f77-72ad-4729-a4e7-bf542d69b142"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.236527    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-podinfo" pod "19331f3a-39f8-46af-8b24-ccfa677abecf" (UID: "19331f3a-39f8-46af-8b24-ccfa677abecf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.237803    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-envoy" pod "cc360f77-72ad-4729-a4e7-bf542d69b142" (UID: "cc360f77-72ad-4729-a4e7-bf542d69b142"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.237840    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19331f3a-39f8-46af-8b24-ccfa677abecf-kube-api-access-g9pp2" pod "19331f3a-39f8-46af-8b24-ccfa677abecf" (UID: "19331f3a-39f8-46af-8b24-ccfa677abecf"). InnerVolumeSpecName "kube-api-access-g9pp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.250086    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-envoy" pod "19331f3a-39f8-46af-8b24-ccfa677abecf" (UID: "19331f3a-39f8-46af-8b24-ccfa677abecf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.339106    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9pp2\" (UniqueName: \"kubernetes.io/projected/19331f3a-39f8-46af-8b24-ccfa677abecf-kube-api-access-g9pp2\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.339135    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc360f77-72ad-4729-a4e7-bf542d69b142-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.339157    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.339170    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.339184    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19331f3a-39f8-46af-8b24-ccfa677abecf-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.339198    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19331f3a-39f8-46af-8b24-ccfa677abecf-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.339211    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.339226    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc360f77-72ad-4729-a4e7-bf542d69b142-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.339238    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc360f77-72ad-4729-a4e7-bf542d69b142-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.340148    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"]
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.340968    6099 cpu_manager.go:414] "containerMap: removing container" podUID="cc360f77-72ad-4729-a4e7-bf542d69b142" containerName="istio-proxy"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.340983    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc360f77-72ad-4729-a4e7-bf542d69b142" containerName="istio-proxy"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.341011    6099 cpu_manager.go:414] "containerMap: removing container" podUID="19331f3a-39f8-46af-8b24-ccfa677abecf" containerName="istio-proxy"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.341017    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19331f3a-39f8-46af-8b24-ccfa677abecf" containerName="istio-proxy"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.341029    6099 cpu_manager.go:414] "containerMap: removing container" podUID="cc360f77-72ad-4729-a4e7-bf542d69b142" containerName="istio-proxy"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.341033    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc360f77-72ad-4729-a4e7-bf542d69b142" containerName="istio-proxy"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.341062    6099 cpu_manager.go:414] "containerMap: removing container" podUID="bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7" containerName="main"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.341067    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7" containerName="main"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.341184    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="19331f3a-39f8-46af-8b24-ccfa677abecf" containerName="istio-proxy"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.341192    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc360f77-72ad-4729-a4e7-bf542d69b142" containerName="istio-proxy"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.341200    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="19331f3a-39f8-46af-8b24-ccfa677abecf" containerName="istio-proxy"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.341210    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd0fdc4-3ea8-45ed-b05a-8f3b35bdd3a7" containerName="main"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.341221    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc360f77-72ad-4729-a4e7-bf542d69b142" containerName="istio-proxy"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.362117    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.375921    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"]
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.441015    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.441097    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.441218    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.441350    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk9ss\" (UniqueName: \"kubernetes.io/projected/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-kube-api-access-mk9ss\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.441408    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.441450    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.441539    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.441606    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.446744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.441810    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.543768 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.543695    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.543912 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.543897    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.543977 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.543959    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.544020 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.543994    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.544073 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.544027    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.544178 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.544156    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk9ss\" (UniqueName: \"kubernetes.io/projected/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-kube-api-access-mk9ss\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.544258 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.544202    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.544326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.544259    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.544326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.544308    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.544437 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.544348    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.544437 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.544382    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.544762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.544631    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.544870 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.544808    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.544870 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.544814    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.546181 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.546156    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.546983 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.546956    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.567171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.567141    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk9ss\" (UniqueName: \"kubernetes.io/projected/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-kube-api-access-mk9ss\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.568651 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.568627    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.694911 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.694882    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:28.888882 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.888847    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"
Oct 30 01:12:28.888882 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.888864    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w" event={"ID":"cc360f77-72ad-4729-a4e7-bf542d69b142","Type":"ContainerDied","Data":"2dbdec5685cbd0c6cbbd85b10ee336bba72df5a2f71670c800abe98d94754c50"}
Oct 30 01:12:28.889407 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.888934    6099 scope.go:122] "RemoveContainer" containerID="3fd6fcdc43306ca714f166a92425d2fa4f40230e6bf49fe722d7600d0df5f4c4"
Oct 30 01:12:28.890690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.890671    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"
Oct 30 01:12:28.890815 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.890720    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p" event={"ID":"19331f3a-39f8-46af-8b24-ccfa677abecf","Type":"ContainerDied","Data":"dc669d150df2112318239373b6097ed1fd69235f9f7b2c4f382475eb3f1c98fa"}
Oct 30 01:12:28.892935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.892911    6099 generic.go:356] "Generic (PLEG): container finished" podID="cc28ff31-b61a-4d8e-9ff3-361345eb9c06" containerID="023512b0f3b8ca26813352bab4f7f74e5b2c877a51d9d733976ef77ae1fd5f58" exitCode=0
Oct 30 01:12:28.893103 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.893023    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7" event={"ID":"cc28ff31-b61a-4d8e-9ff3-361345eb9c06","Type":"ContainerDied","Data":"023512b0f3b8ca26813352bab4f7f74e5b2c877a51d9d733976ef77ae1fd5f58"}
Oct 30 01:12:28.893662 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.893591    6099 scope.go:122] "RemoveContainer" containerID="023512b0f3b8ca26813352bab4f7f74e5b2c877a51d9d733976ef77ae1fd5f58"
Oct 30 01:12:28.895377 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.895342    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:12:28.895496 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.895442    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:12:28.895496 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.895478    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:12:28.902568 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.902554    6099 scope.go:122] "RemoveContainer" containerID="244f27a398e1fa99891444d79abe5b6c0ab86d6a62e429fdadc21e37bebbb848"
Oct 30 01:12:28.988350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.988322    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"]
Oct 30 01:12:28.995378 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:28.995349    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-99w7w"]
Oct 30 01:12:29.028758 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:29.026928    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"]
Oct 30 01:12:29.034620 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:29.034579    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:12:29.034791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:29.034664    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:12:29.034791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:29.034723    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:12:29.035859 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:29.035813    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"]
Oct 30 01:12:29.041205 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:29.041181    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5062f6-openshift-default-68b95c9cff-jqv4p"]
Oct 30 01:12:29.948051 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:29.948016    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs" event={"ID":"a528dfda-e8fb-4a8e-abe7-4d870b94a64a","Type":"ContainerStarted","Data":"865c196c81a38de3defa6e460c36ec447807e39d28e0b7c2ce9155138c26e0f0"}
Oct 30 01:12:29.948463 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:29.948069    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs" event={"ID":"a528dfda-e8fb-4a8e-abe7-4d870b94a64a","Type":"ContainerStarted","Data":"854de2b4a9143c279e624e69b4a20ff5ecedf3d7fd689e7761a26c84fbc25a80"}
Oct 30 01:12:29.954854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:29.954828    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7" event={"ID":"cc28ff31-b61a-4d8e-9ff3-361345eb9c06","Type":"ContainerStarted","Data":"afda7568d68e390d0b0a7b46b543e5a2ad087b6448e740fe8221162f1de57775"}
Oct 30 01:12:30.064001 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:30.063953    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs" podStartSLOduration=2.063938657 podStartE2EDuration="2.063938657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:30.01823536 +0000 UTC m=+2243.875143128" watchObservedRunningTime="2026-10-30 01:12:30.063938657 +0000 UTC m=+2243.920846412"
Oct 30 01:12:30.263933 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:30.263859    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19331f3a-39f8-46af-8b24-ccfa677abecf" path="/var/lib/kubelet/pods/19331f3a-39f8-46af-8b24-ccfa677abecf/volumes"
Oct 30 01:12:30.264408 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:30.264393    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc360f77-72ad-4729-a4e7-bf542d69b142" path="/var/lib/kubelet/pods/cc360f77-72ad-4729-a4e7-bf542d69b142/volumes"
Oct 30 01:12:30.361153 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:30.361120    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:12:30.362148 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:30.362125    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body=
Oct 30 01:12:30.362211 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:30.362177    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7" podUID="cc28ff31-b61a-4d8e-9ff3-361345eb9c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused"
Oct 30 01:12:30.449577 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:30.449547    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:12:30.705730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:30.705073    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:30.710729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:30.706036    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Oct 30 01:12:30.710729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:30.706088    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs" podUID="a528dfda-e8fb-4a8e-abe7-4d870b94a64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Oct 30 01:12:31.360577 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:31.360537    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body=
Oct 30 01:12:31.361119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:31.360624    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7" podUID="cc28ff31-b61a-4d8e-9ff3-361345eb9c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused"
Oct 30 01:12:31.709607 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:31.705354    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Oct 30 01:12:31.709607 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:31.705454    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs" podUID="a528dfda-e8fb-4a8e-abe7-4d870b94a64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Oct 30 01:12:32.361740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:32.361066    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body=
Oct 30 01:12:32.361740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:32.361122    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7" podUID="cc28ff31-b61a-4d8e-9ff3-361345eb9c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused"
Oct 30 01:12:32.707778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:32.707245    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Oct 30 01:12:32.707778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:32.707309    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs" podUID="a528dfda-e8fb-4a8e-abe7-4d870b94a64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Oct 30 01:12:33.361202 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:33.361170    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body=
Oct 30 01:12:33.361414 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:33.361394    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7" podUID="cc28ff31-b61a-4d8e-9ff3-361345eb9c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused"
Oct 30 01:12:33.701785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:33.701180    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Oct 30 01:12:33.701785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:33.701242    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs" podUID="a528dfda-e8fb-4a8e-abe7-4d870b94a64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Oct 30 01:12:34.361227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:34.361194    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body=
Oct 30 01:12:34.361560 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:34.361538    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7" podUID="cc28ff31-b61a-4d8e-9ff3-361345eb9c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused"
Oct 30 01:12:34.715851 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:34.704109    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Oct 30 01:12:34.715851 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:34.704171    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs" podUID="a528dfda-e8fb-4a8e-abe7-4d870b94a64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Oct 30 01:12:35.365358 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:35.365323    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:12:35.366451 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:35.366426    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:12:35.699881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:35.699807    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:35.700111 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:35.700085    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:35.700839 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:35.700822    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:12:41.361945 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.361908    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"]
Oct 30 01:12:41.363146 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.363112    6099 cpu_manager.go:414] "containerMap: removing container" podUID="19331f3a-39f8-46af-8b24-ccfa677abecf" containerName="istio-proxy"
Oct 30 01:12:41.363146 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.363146    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19331f3a-39f8-46af-8b24-ccfa677abecf" containerName="istio-proxy"
Oct 30 01:12:41.369156 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.369133    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.386244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.386217    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"]
Oct 30 01:12:41.391277 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.391255    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-95492e\"/\"default-dockercfg-9gpmj\""
Oct 30 01:12:41.391396 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.391312    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-95492e\"/\"sub-test-model-95492e-kserve-self-signed-certs\""
Oct 30 01:12:41.395447 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.395426    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-95492e\"/\"openshift-service-ca.crt\""
Oct 30 01:12:41.395621 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.395602    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-95492e\"/\"kube-root-ca.crt\""
Oct 30 01:12:41.416781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.416752    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e36adae7-03ed-4647-9410-5b4bceafa593-tls-certs\") pod \"sub-test-model-95492e-kserve-84dc465d9c-lgcxn\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") " pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.416917 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.416806    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-home\") pod \"sub-test-model-95492e-kserve-84dc465d9c-lgcxn\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") " pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.416917 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.416847    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-tmp-dir\") pod \"sub-test-model-95492e-kserve-84dc465d9c-lgcxn\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") " pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.417088 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.417059    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-model-cache\") pod \"sub-test-model-95492e-kserve-84dc465d9c-lgcxn\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") " pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.417208 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.417185    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r4mj\" (UniqueName: \"kubernetes.io/projected/e36adae7-03ed-4647-9410-5b4bceafa593-kube-api-access-9r4mj\") pod \"sub-test-model-95492e-kserve-84dc465d9c-lgcxn\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") " pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.417276 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.417258    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-dshm\") pod \"sub-test-model-95492e-kserve-84dc465d9c-lgcxn\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") " pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.519091 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.519058    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-model-cache\") pod \"sub-test-model-95492e-kserve-84dc465d9c-lgcxn\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") " pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.519264 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.519149    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r4mj\" (UniqueName: \"kubernetes.io/projected/e36adae7-03ed-4647-9410-5b4bceafa593-kube-api-access-9r4mj\") pod \"sub-test-model-95492e-kserve-84dc465d9c-lgcxn\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") " pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.519264 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.519195    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-dshm\") pod \"sub-test-model-95492e-kserve-84dc465d9c-lgcxn\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") " pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.519409 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.519310    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e36adae7-03ed-4647-9410-5b4bceafa593-tls-certs\") pod \"sub-test-model-95492e-kserve-84dc465d9c-lgcxn\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") " pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.519409 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.519332    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-home\") pod \"sub-test-model-95492e-kserve-84dc465d9c-lgcxn\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") " pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.519409 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.519356    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-tmp-dir\") pod \"sub-test-model-95492e-kserve-84dc465d9c-lgcxn\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") " pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.519576 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.519512    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-model-cache\") pod \"sub-test-model-95492e-kserve-84dc465d9c-lgcxn\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") " pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.519731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.519690    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-home\") pod \"sub-test-model-95492e-kserve-84dc465d9c-lgcxn\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") " pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.519805 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.519762    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-tmp-dir\") pod \"sub-test-model-95492e-kserve-84dc465d9c-lgcxn\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") " pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.521759 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.521731    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-dshm\") pod \"sub-test-model-95492e-kserve-84dc465d9c-lgcxn\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") " pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.521884 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.521839    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e36adae7-03ed-4647-9410-5b4bceafa593-tls-certs\") pod \"sub-test-model-95492e-kserve-84dc465d9c-lgcxn\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") " pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.541985 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.541963    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r4mj\" (UniqueName: \"kubernetes.io/projected/e36adae7-03ed-4647-9410-5b4bceafa593-kube-api-access-9r4mj\") pod \"sub-test-model-95492e-kserve-84dc465d9c-lgcxn\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") " pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.687164 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.687071    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:41.997901 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:12:41.997869    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode36adae7_03ed_4647_9410_5b4bceafa593.slice/crio-6796a70404cb700bd4cda68be2f0644d56ab5b837f8ddd7ec6a66e7e2e878bf0 WatchSource:0}: Error finding container 6796a70404cb700bd4cda68be2f0644d56ab5b837f8ddd7ec6a66e7e2e878bf0: Status 404 returned error can't find the container with id 6796a70404cb700bd4cda68be2f0644d56ab5b837f8ddd7ec6a66e7e2e878bf0
Oct 30 01:12:41.999039 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:41.999011    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"]
Oct 30 01:12:42.118044 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:42.117177    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn" event={"ID":"e36adae7-03ed-4647-9410-5b4bceafa593","Type":"ContainerStarted","Data":"0dffec5ca437a3adfb0d19315491859e85503c5bffc793915778c8cea386157b"}
Oct 30 01:12:42.118044 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:42.117228    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn" event={"ID":"e36adae7-03ed-4647-9410-5b4bceafa593","Type":"ContainerStarted","Data":"6796a70404cb700bd4cda68be2f0644d56ab5b837f8ddd7ec6a66e7e2e878bf0"}
Oct 30 01:12:42.160901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:42.160825    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn" podStartSLOduration=1.160803891 podStartE2EDuration="1.160803891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:42.157977423 +0000 UTC m=+2256.014885179" watchObservedRunningTime="2026-10-30 01:12:42.160803891 +0000 UTC m=+2256.017711648"
Oct 30 01:12:42.687821 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:42.687792    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:46.630719 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:12:46.630650    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:12:51.687871 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:51.687835    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:51.700936 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:51.700907    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:52.231622 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:52.231585    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:12:55.203458 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:55.203417    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:12:57.752038 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:57.743832    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"]
Oct 30 01:12:57.769674 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:57.769643    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:57.785855 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:57.785806    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmr7z\" (UniqueName: \"kubernetes.io/projected/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-kube-api-access-jmr7z\") pod \"auth-test-model-d01638-kserve-559c6d4469-grrdf\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") " pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:57.786200 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:57.786170    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-tls-certs\") pod \"auth-test-model-d01638-kserve-559c6d4469-grrdf\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") " pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:57.786418 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:57.786399    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-model-cache\") pod \"auth-test-model-d01638-kserve-559c6d4469-grrdf\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") " pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:57.786654 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:57.786630    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-home\") pod \"auth-test-model-d01638-kserve-559c6d4469-grrdf\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") " pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:57.786914 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:57.786894    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-tmp-dir\") pod \"auth-test-model-d01638-kserve-559c6d4469-grrdf\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") " pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:57.787133 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:57.787113    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-dshm\") pod \"auth-test-model-d01638-kserve-559c6d4469-grrdf\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") " pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:57.804458 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:57.804402    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"]
Oct 30 01:12:57.843738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:57.813240    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-d01638\"/\"default-dockercfg-l4wc4\""
Oct 30 01:12:57.843738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:57.813337    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-d01638\"/\"auth-test-model-d01638-kserve-self-signed-certs\""
Oct 30 01:12:57.843738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:57.813464    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-d01638\"/\"kube-root-ca.crt\""
Oct 30 01:12:57.843738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:57.813521    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-d01638\"/\"openshift-service-ca.crt\""
Oct 30 01:12:58.157730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:58.153917    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-tmp-dir\") pod \"auth-test-model-d01638-kserve-559c6d4469-grrdf\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") " pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:58.172736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:58.165849    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-dshm\") pod \"auth-test-model-d01638-kserve-559c6d4469-grrdf\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") " pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:58.185731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:58.173161    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmr7z\" (UniqueName: \"kubernetes.io/projected/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-kube-api-access-jmr7z\") pod \"auth-test-model-d01638-kserve-559c6d4469-grrdf\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") " pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:58.259936 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:58.259893    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-tls-certs\") pod \"auth-test-model-d01638-kserve-559c6d4469-grrdf\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") " pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:58.260261 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:58.260237    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-model-cache\") pod \"auth-test-model-d01638-kserve-559c6d4469-grrdf\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") " pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:58.260507 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:58.260489    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-home\") pod \"auth-test-model-d01638-kserve-559c6d4469-grrdf\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") " pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:58.261165 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:58.261139    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-home\") pod \"auth-test-model-d01638-kserve-559c6d4469-grrdf\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") " pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:58.261330 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:58.163118    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-tmp-dir\") pod \"auth-test-model-d01638-kserve-559c6d4469-grrdf\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") " pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:58.264171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:58.264146    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-model-cache\") pod \"auth-test-model-d01638-kserve-559c6d4469-grrdf\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") " pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:58.297027 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:58.296992    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-dshm\") pod \"auth-test-model-d01638-kserve-559c6d4469-grrdf\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") " pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:58.307825 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:58.298363    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-tls-certs\") pod \"auth-test-model-d01638-kserve-559c6d4469-grrdf\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") " pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:58.308078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:58.304372    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmr7z\" (UniqueName: \"kubernetes.io/projected/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-kube-api-access-jmr7z\") pod \"auth-test-model-d01638-kserve-559c6d4469-grrdf\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") " pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:58.417362 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:58.417266    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:12:58.965454 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:58.965337    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"]
Oct 30 01:12:59.457032 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:59.456993    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf" event={"ID":"5f0f160f-9861-48e1-8a8d-6e33f9a46a46","Type":"ContainerStarted","Data":"4fd468433f433cb2dd2aa01f723359425c5cbc596275ac168dd98061a6dddd2b"}
Oct 30 01:12:59.457310 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:59.457280    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf" event={"ID":"5f0f160f-9861-48e1-8a8d-6e33f9a46a46","Type":"ContainerStarted","Data":"f6f45d2ab2939bd50d725cc05ca65a719a3f1aa66b3a74f00ad383b9f16383f1"}
Oct 30 01:12:59.531831 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:12:59.531786    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf" podStartSLOduration=2.531773335 podStartE2EDuration="2.531773335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:59.527284086 +0000 UTC m=+2273.384191840" watchObservedRunningTime="2026-10-30 01:12:59.531773335 +0000 UTC m=+2273.388681090"
Oct 30 01:13:00.419161 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:00.419132    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:13:00.817518 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:00.817482    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"]
Oct 30 01:13:00.823230 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:00.823204    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:13:00.852053 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:00.850131    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-f69104-serving-cert\""
Oct 30 01:13:00.857261 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:00.857236    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"]
Oct 30 01:13:00.946613 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:00.946570    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7\" (UID: \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:13:00.946818 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:00.946756    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7\" (UID: \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:13:00.946951 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:00.946923    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmt6j\" (UniqueName: \"kubernetes.io/projected/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-kube-api-access-qmt6j\") pod \"maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7\" (UID: \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:13:00.947079 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:00.947060    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7\" (UID: \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:13:01.048499 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:01.048464    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmt6j\" (UniqueName: \"kubernetes.io/projected/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-kube-api-access-qmt6j\") pod \"maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7\" (UID: \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:13:01.048699 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:01.048556    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7\" (UID: \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:13:01.048699 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:01.048682    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7\" (UID: \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:13:01.048889 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:13:01.048786    6099 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-serving-cert: secret "maas-api-e2e-shared-b-w3-f69104-serving-cert" not found
Oct 30 01:13:01.048889 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:13:01.048854    6099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-maas-api-tls podName:7fb31e16-f069-4549-ab6b-c8cdb62b43aa nodeName:}" failed. No retries permitted until 2026-10-30 01:13:01.54883342 +0000 UTC m=+2275.405741171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-maas-api-tls") pod "maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7" (UID: "7fb31e16-f069-4549-ab6b-c8cdb62b43aa") : secret "maas-api-e2e-shared-b-w3-f69104-serving-cert" not found
Oct 30 01:13:01.049046 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:01.048996    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7\" (UID: \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:13:01.049831 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:01.049806    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7\" (UID: \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:13:01.052142 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:01.052119    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7\" (UID: \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:13:01.074524 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:01.074458    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmt6j\" (UniqueName: \"kubernetes.io/projected/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-kube-api-access-qmt6j\") pod \"maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7\" (UID: \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:13:01.557359 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:01.557321    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7\" (UID: \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:13:01.572682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:01.572652    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7\" (UID: \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:13:01.740221 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:01.740184    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:13:02.062558 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:13:02.062529    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb31e16_f069_4549_ab6b_c8cdb62b43aa.slice/crio-58562a4494edf45e1f72bd7cf1987f5bbb72e36a209647132122733d574ccf2a WatchSource:0}: Error finding container 58562a4494edf45e1f72bd7cf1987f5bbb72e36a209647132122733d574ccf2a: Status 404 returned error can't find the container with id 58562a4494edf45e1f72bd7cf1987f5bbb72e36a209647132122733d574ccf2a
Oct 30 01:13:02.062865 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:02.062842    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"]
Oct 30 01:13:02.491982 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:02.487685    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7" event={"ID":"7fb31e16-f069-4549-ab6b-c8cdb62b43aa","Type":"ContainerStarted","Data":"58562a4494edf45e1f72bd7cf1987f5bbb72e36a209647132122733d574ccf2a"}
Oct 30 01:13:03.501378 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:03.501338    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7" event={"ID":"7fb31e16-f069-4549-ab6b-c8cdb62b43aa","Type":"ContainerStarted","Data":"80f91f36be7d5ed36785d19b031597ee600a2de93a5b2f5adc7d7ca026b987d3"}
Oct 30 01:13:03.501882 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:03.501441    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:13:03.551415 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:03.551361    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7" podStartSLOduration=3.04797139 podStartE2EDuration="3.551347274s" totalImagesPullingTime="503.375884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:03.5492073 +0000 UTC m=+2277.406115056" watchObservedRunningTime="2026-10-30 01:13:03.551347274 +0000 UTC m=+2277.408255028"
Oct 30 01:13:08.419158 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:08.419124    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:13:08.432290 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:08.432266    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:13:08.560371 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:08.560345    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:13:09.515468 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:09.515439    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:13:11.030722 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:11.030678    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h"]
Oct 30 01:13:11.042086 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:11.042061    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba1483c0-ddg2h"]
Oct 30 01:13:12.265692 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:12.265650    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1142c8f9-ffd0-4231-844c-d6236e976047" path="/var/lib/kubelet/pods/1142c8f9-ffd0-4231-844c-d6236e976047/volumes"
Oct 30 01:13:13.494194 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.494160    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"]
Oct 30 01:13:13.503183 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.503158    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.546726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.542837    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"]
Oct 30 01:13:13.570901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.570862    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.571453 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.570916    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49482cc2-07e6-455e-952e-bd42810e8d7b-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.571453 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.570975    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gfh5\" (UniqueName: \"kubernetes.io/projected/49482cc2-07e6-455e-952e-bd42810e8d7b-kube-api-access-2gfh5\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.571453 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.571003    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.571453 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.571049    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.571453 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.571077    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.571453 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.571341    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.571453 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.571443    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.571927 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.571519    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.672761 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.672645    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.672958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.672761    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.672958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.672820    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.673114 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.672979    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.673114 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.673024    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49482cc2-07e6-455e-952e-bd42810e8d7b-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.673114 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.673081    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gfh5\" (UniqueName: \"kubernetes.io/projected/49482cc2-07e6-455e-952e-bd42810e8d7b-kube-api-access-2gfh5\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.673114 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.673109    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.673338 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.673154    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.673338 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.673155    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.673338 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.673182    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.673338 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.673253    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.673338 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.673308    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.673649 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.673594    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.673785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.673768    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49482cc2-07e6-455e-952e-bd42810e8d7b-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.675188 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.675165    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.675689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.675670    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.696328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.696295    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.696429 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.696412    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gfh5\" (UniqueName: \"kubernetes.io/projected/49482cc2-07e6-455e-952e-bd42810e8d7b-kube-api-access-2gfh5\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:13.835261 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:13.835227    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:14.161949 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:13:14.161917    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49482cc2_07e6_455e_952e_bd42810e8d7b.slice/crio-8994b89f6ae26e1b7eca222933af3e7e89d73b9fce4235e9b30a11f9d70c3a92 WatchSource:0}: Error finding container 8994b89f6ae26e1b7eca222933af3e7e89d73b9fce4235e9b30a11f9d70c3a92: Status 404 returned error can't find the container with id 8994b89f6ae26e1b7eca222933af3e7e89d73b9fce4235e9b30a11f9d70c3a92
Oct 30 01:13:14.162970 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:14.162947    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"]
Oct 30 01:13:14.164885 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:14.164852    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:13:14.164959 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:14.164930    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:13:14.165009 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:14.164960    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:13:14.604467 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:14.604426    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l" event={"ID":"49482cc2-07e6-455e-952e-bd42810e8d7b","Type":"ContainerStarted","Data":"bada115a463ac0a8ce34cd05d86de2cbfae7bd94d16fece22f9e8874a3d98c8c"}
Oct 30 01:13:14.604881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:14.604495    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l" event={"ID":"49482cc2-07e6-455e-952e-bd42810e8d7b","Type":"ContainerStarted","Data":"8994b89f6ae26e1b7eca222933af3e7e89d73b9fce4235e9b30a11f9d70c3a92"}
Oct 30 01:13:14.650882 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:14.650826    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l" podStartSLOduration=1.6508114090000001 podStartE2EDuration="1.650811409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:14.648877571 +0000 UTC m=+2288.505785351" watchObservedRunningTime="2026-10-30 01:13:14.650811409 +0000 UTC m=+2288.507719154"
Oct 30 01:13:14.835471 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:14.835432    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:15.032024 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:15.031940    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p"]
Oct 30 01:13:15.036273 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:15.036249    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0f2044-54t2p"]
Oct 30 01:13:15.837160 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:15.837125    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Oct 30 01:13:15.837549 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:15.837191    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l" podUID="49482cc2-07e6-455e-952e-bd42810e8d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Oct 30 01:13:16.293976 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:16.293943    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b105e1-3566-447d-bfc8-f250043b8341" path="/var/lib/kubelet/pods/31b105e1-3566-447d-bfc8-f250043b8341/volumes"
Oct 30 01:13:16.632330 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:13:16.632253    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:13:16.836180 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:16.836010    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Oct 30 01:13:16.836389 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:16.836297    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l" podUID="49482cc2-07e6-455e-952e-bd42810e8d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Oct 30 01:13:17.835923 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:17.835888    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Oct 30 01:13:17.836297 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:17.835942    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l" podUID="49482cc2-07e6-455e-952e-bd42810e8d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Oct 30 01:13:18.836087 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:18.836052    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Oct 30 01:13:18.836496 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:18.836104    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l" podUID="49482cc2-07e6-455e-952e-bd42810e8d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Oct 30 01:13:19.836090 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:19.836051    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Oct 30 01:13:19.836269 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:19.836121    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l" podUID="49482cc2-07e6-455e-952e-bd42810e8d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Oct 30 01:13:20.841335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:20.841302    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:20.841855 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:20.841833    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:20.842473 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:20.842451    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:13:27.034129 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:27.034093    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd"]
Oct 30 01:13:27.042886 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:27.042857    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ffb27d0-gpjqd"]
Oct 30 01:13:28.265471 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:28.265438    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa50df9-7fa7-48ac-a05b-3cd783aee9da" path="/var/lib/kubelet/pods/7aa50df9-7fa7-48ac-a05b-3cd783aee9da/volumes"
Oct 30 01:13:37.837891 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:37.837843    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:13:37.838420 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:37.837943    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz" podUID="e2c83697-c779-4359-804d-b6b4b4d5e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:38.211059 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:38.210953    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"]
Oct 30 01:13:39.107898 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.107865    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"]
Oct 30 01:13:39.127160 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.127129    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.144291 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.144257    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"]
Oct 30 01:13:39.148062 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.148029    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f91e87-openshift-default-dockercfg-6k6mt\""
Oct 30 01:13:39.160008 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.159980    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-data\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.160160 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.160042    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-workload-socket\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.160160 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.160115    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-token\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.160288 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.160172    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-envoy\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.160288 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.160257    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-workload-certs\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.160400 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.160368    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istiod-ca-cert\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.161255 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.161229    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hqkd\" (UniqueName: \"kubernetes.io/projected/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-kube-api-access-7hqkd\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.161370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.161322    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-credential-socket\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.161370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.161349    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-podinfo\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.262997 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.262956    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-workload-certs\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.263204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.263077    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istiod-ca-cert\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.263204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.263137    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hqkd\" (UniqueName: \"kubernetes.io/projected/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-kube-api-access-7hqkd\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.263347 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.263203    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-credential-socket\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.263459 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.263331    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-podinfo\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.263459 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.263357    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-workload-certs\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.263588 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.263552    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-data\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.263648 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.263576    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-credential-socket\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.263648 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.263593    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-workload-socket\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.263849 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.263691    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-token\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.263849 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.263806    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-envoy\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.263849 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.263819    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istiod-ca-cert\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.264028 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.263870    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-workload-socket\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.264028 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.263893    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-data\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.286005 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.285972    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-envoy\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.286236 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.286215    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-podinfo\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.286942 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.286917    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hqkd\" (UniqueName: \"kubernetes.io/projected/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-kube-api-access-7hqkd\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.288215 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.288196    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-token\") pod \"e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") " pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:39.508782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:39.494111    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:40.032222 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:40.032189    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8"]
Oct 30 01:13:40.042813 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:40.042782    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361af8cf-75zz8"]
Oct 30 01:13:40.265464 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:40.265405    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="851b0e85-b868-4d95-9ffa-ccf46d2cf31b" path="/var/lib/kubelet/pods/851b0e85-b868-4d95-9ffa-ccf46d2cf31b/volumes"
Oct 30 01:13:40.607107 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:40.607029    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"]
Oct 30 01:13:40.618025 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:13:40.617998    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8655eb2_c236_49f0_90e8_b7b4ee6e51be.slice/crio-8c299ccd5347c8b4048098d0918125326aad89dd2eb970aabbbef36a950f55da WatchSource:0}: Error finding container 8c299ccd5347c8b4048098d0918125326aad89dd2eb970aabbbef36a950f55da: Status 404 returned error can't find the container with id 8c299ccd5347c8b4048098d0918125326aad89dd2eb970aabbbef36a950f55da
Oct 30 01:13:40.620923 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:40.620890    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:13:40.621030 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:40.620994    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:13:40.621074 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:40.621028    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:13:40.892993 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:40.892914    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m" event={"ID":"d8655eb2-c236-49f0-90e8-b7b4ee6e51be","Type":"ContainerStarted","Data":"8c299ccd5347c8b4048098d0918125326aad89dd2eb970aabbbef36a950f55da"}
Oct 30 01:13:41.904609 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:41.904564    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m" event={"ID":"d8655eb2-c236-49f0-90e8-b7b4ee6e51be","Type":"ContainerStarted","Data":"b75dd08ca618e2049d074d706ebdf6720db7fd88718488d2f74f404fdb7c5c66"}
Oct 30 01:13:41.965352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:41.965291    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m" podStartSLOduration=2.965274462 podStartE2EDuration="2.965274462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:41.960379848 +0000 UTC m=+2315.817287615" watchObservedRunningTime="2026-10-30 01:13:41.965274462 +0000 UTC m=+2315.822182216"
Oct 30 01:13:42.499933 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:42.499895    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:42.505938 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:42.505909    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:42.913438 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:42.913409    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:42.914415 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:42.914391    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:13:45.132076 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:45.132041    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"]
Oct 30 01:13:46.718109 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:13:46.718068    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:13:46.967138 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:46.967103    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j"]
Oct 30 01:13:46.984827 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:46.984743    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j"
Oct 30 01:13:46.991443 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:46.991409    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j"]
Oct 30 01:13:47.012636 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:47.012612    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f91e87\""
Oct 30 01:13:47.145731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:47.145428    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67wlb\" (UniqueName: \"kubernetes.io/projected/40f9d853-515b-48e7-b72b-5fd68d43ac4f-kube-api-access-67wlb\") pod \"payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j\" (UID: \"40f9d853-515b-48e7-b72b-5fd68d43ac4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j"
Oct 30 01:13:47.145731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:47.145501    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f9d853-515b-48e7-b72b-5fd68d43ac4f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j\" (UID: \"40f9d853-515b-48e7-b72b-5fd68d43ac4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j"
Oct 30 01:13:47.249781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:47.249466    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67wlb\" (UniqueName: \"kubernetes.io/projected/40f9d853-515b-48e7-b72b-5fd68d43ac4f-kube-api-access-67wlb\") pod \"payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j\" (UID: \"40f9d853-515b-48e7-b72b-5fd68d43ac4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j"
Oct 30 01:13:47.249781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:47.249547    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f9d853-515b-48e7-b72b-5fd68d43ac4f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j\" (UID: \"40f9d853-515b-48e7-b72b-5fd68d43ac4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j"
Oct 30 01:13:47.253731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:47.250613    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f9d853-515b-48e7-b72b-5fd68d43ac4f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j\" (UID: \"40f9d853-515b-48e7-b72b-5fd68d43ac4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j"
Oct 30 01:13:47.312993 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:47.307052    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67wlb\" (UniqueName: \"kubernetes.io/projected/40f9d853-515b-48e7-b72b-5fd68d43ac4f-kube-api-access-67wlb\") pod \"payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j\" (UID: \"40f9d853-515b-48e7-b72b-5fd68d43ac4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j"
Oct 30 01:13:47.325725 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:47.316045    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j"
Oct 30 01:13:47.757578 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:13:47.757496    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f9d853_515b_48e7_b72b_5fd68d43ac4f.slice/crio-bbfd2d6bfc45320ee93e72563e681c6a2e9518ee2220972f8ca50e60efe0a783 WatchSource:0}: Error finding container bbfd2d6bfc45320ee93e72563e681c6a2e9518ee2220972f8ca50e60efe0a783: Status 404 returned error can't find the container with id bbfd2d6bfc45320ee93e72563e681c6a2e9518ee2220972f8ca50e60efe0a783
Oct 30 01:13:47.759811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:47.759778    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j"]
Oct 30 01:13:47.969112 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:47.969074    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j" event={"ID":"40f9d853-515b-48e7-b72b-5fd68d43ac4f","Type":"ContainerStarted","Data":"50357b7fb72e57c1c330b222664b98ea5490e8ab14551d2d4b1a9d9097d11f9e"}
Oct 30 01:13:47.969286 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:47.969172    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j"
Oct 30 01:13:47.969286 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:47.969199    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j" event={"ID":"40f9d853-515b-48e7-b72b-5fd68d43ac4f","Type":"ContainerStarted","Data":"bbfd2d6bfc45320ee93e72563e681c6a2e9518ee2220972f8ca50e60efe0a783"}
Oct 30 01:13:48.018655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:48.018568    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j" podStartSLOduration=2.018555466 podStartE2EDuration="2.018555466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:48.014529662 +0000 UTC m=+2321.871437418" watchObservedRunningTime="2026-10-30 01:13:48.018555466 +0000 UTC m=+2321.875463221"
Oct 30 01:13:49.989240 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:49.989210    6099 generic.go:356] "Generic (PLEG): container finished" podID="e2c83697-c779-4359-804d-b6b4b4d5e77c" containerID="1f43de5288b1add050adc94449f17f9bcfbad724425c6e912e546a11f767230c" exitCode=0
Oct 30 01:13:49.989664 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:49.989285    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz" event={"ID":"e2c83697-c779-4359-804d-b6b4b4d5e77c","Type":"ContainerDied","Data":"1f43de5288b1add050adc94449f17f9bcfbad724425c6e912e546a11f767230c"}
Oct 30 01:13:49.989801 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:49.989786    6099 scope.go:122] "RemoveContainer" containerID="1f43de5288b1add050adc94449f17f9bcfbad724425c6e912e546a11f767230c"
Oct 30 01:13:49.991778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:49.991746    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:13:49.991884 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:49.991825    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:13:49.991884 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:49.991859    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:13:51.000657 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:51.000624    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz" event={"ID":"e2c83697-c779-4359-804d-b6b4b4d5e77c","Type":"ContainerStarted","Data":"ea435deb2d0c40088ab9a8815b3d6f6a76b4cb8e74a297a9ecf99118259db5e6"}
Oct 30 01:13:51.786731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:51.764722    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:13:51.789908 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:51.789868    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Oct 30 01:13:51.790066 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:51.789931    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz" podUID="e2c83697-c779-4359-804d-b6b4b4d5e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Oct 30 01:13:51.797001 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:51.796960    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:13:52.032460 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:52.032425    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp"]
Oct 30 01:13:52.043911 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:52.043888    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9656d13e-bm7bp"]
Oct 30 01:13:52.268731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:52.264914    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89fa8089-4f05-4736-93c5-42f3362cf38c" path="/var/lib/kubelet/pods/89fa8089-4f05-4736-93c5-42f3362cf38c/volumes"
Oct 30 01:13:52.768733 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:52.764772    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Oct 30 01:13:52.768733 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:52.764831    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz" podUID="e2c83697-c779-4359-804d-b6b4b4d5e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Oct 30 01:13:53.542130 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:53.542089    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb" podUID="30fbfcf0-f18a-4949-92d8-fbbb726ed62d" containerName="main" containerID="cri-o://f50edf569e1c2a0f9e3f3376c29bb5374e252c00ae6839551dc0ced0a59bfe16" gracePeriod=45
Oct 30 01:13:53.795834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:53.795186    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Oct 30 01:13:53.795834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:53.795251    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz" podUID="e2c83697-c779-4359-804d-b6b4b4d5e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Oct 30 01:13:53.907530 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:53.907496    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:13:53.998231 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:53.998199    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-dshm\") pod \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") "
Oct 30 01:13:53.998532 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:53.998507    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-tmp-dir\") pod \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") "
Oct 30 01:13:53.998788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:53.998769    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-home\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-home\") pod \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") "
Oct 30 01:13:53.999000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:53.998980    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-kube-api-access-bt9vs\" (UniqueName: \"kubernetes.io/projected/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-kube-api-access-bt9vs\") pod \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") "
Oct 30 01:13:53.999192 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:53.999175    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-model-cache\") pod \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") "
Oct 30 01:13:53.999357 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:53.999341    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-tls-certs\" (UniqueName: \"kubernetes.io/secret/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-tls-certs\") pod \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\" (UID: \"30fbfcf0-f18a-4949-92d8-fbbb726ed62d\") "
Oct 30 01:13:53.999897 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:53.999869    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-home" pod "30fbfcf0-f18a-4949-92d8-fbbb726ed62d" (UID: "30fbfcf0-f18a-4949-92d8-fbbb726ed62d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:54.000463 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.000434    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-model-cache" pod "30fbfcf0-f18a-4949-92d8-fbbb726ed62d" (UID: "30fbfcf0-f18a-4949-92d8-fbbb726ed62d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:54.000735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.000699    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:13:54.000861 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.000847    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:13:54.000986 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.000923    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-tmp-dir" pod "30fbfcf0-f18a-4949-92d8-fbbb726ed62d" (UID: "30fbfcf0-f18a-4949-92d8-fbbb726ed62d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:54.004188 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.003827    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-dshm" pod "30fbfcf0-f18a-4949-92d8-fbbb726ed62d" (UID: "30fbfcf0-f18a-4949-92d8-fbbb726ed62d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:54.004188 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.004134    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-tls-certs" pod "30fbfcf0-f18a-4949-92d8-fbbb726ed62d" (UID: "30fbfcf0-f18a-4949-92d8-fbbb726ed62d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:54.010193 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.010165    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-kube-api-access-bt9vs" pod "30fbfcf0-f18a-4949-92d8-fbbb726ed62d" (UID: "30fbfcf0-f18a-4949-92d8-fbbb726ed62d"). InnerVolumeSpecName "kube-api-access-bt9vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:54.029162 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.029135    6099 generic.go:356] "Generic (PLEG): container finished" podID="30fbfcf0-f18a-4949-92d8-fbbb726ed62d" containerID="f50edf569e1c2a0f9e3f3376c29bb5374e252c00ae6839551dc0ced0a59bfe16" exitCode=0
Oct 30 01:13:54.029236 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.029187    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"
Oct 30 01:13:54.029236 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.029199    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb" event={"ID":"30fbfcf0-f18a-4949-92d8-fbbb726ed62d","Type":"ContainerDied","Data":"f50edf569e1c2a0f9e3f3376c29bb5374e252c00ae6839551dc0ced0a59bfe16"}
Oct 30 01:13:54.029358 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.029270    6099 scope.go:122] "RemoveContainer" containerID="f50edf569e1c2a0f9e3f3376c29bb5374e252c00ae6839551dc0ced0a59bfe16"
Oct 30 01:13:54.029464 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.029445    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb" event={"ID":"30fbfcf0-f18a-4949-92d8-fbbb726ed62d","Type":"ContainerDied","Data":"121115c73ee513e49aa2e7486cda356b0169fb392e3617c8a7a5c50daa533b87"}
Oct 30 01:13:54.059775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.059758    6099 scope.go:122] "RemoveContainer" containerID="f50edf569e1c2a0f9e3f3376c29bb5374e252c00ae6839551dc0ced0a59bfe16"
Oct 30 01:13:54.060056 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:13:54.060039    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f50edf569e1c2a0f9e3f3376c29bb5374e252c00ae6839551dc0ced0a59bfe16\": container with ID starting with f50edf569e1c2a0f9e3f3376c29bb5374e252c00ae6839551dc0ced0a59bfe16 not found: ID does not exist" containerID="f50edf569e1c2a0f9e3f3376c29bb5374e252c00ae6839551dc0ced0a59bfe16"
Oct 30 01:13:54.060114 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.060062    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50edf569e1c2a0f9e3f3376c29bb5374e252c00ae6839551dc0ced0a59bfe16"} err="failed to get container status \"f50edf569e1c2a0f9e3f3376c29bb5374e252c00ae6839551dc0ced0a59bfe16\": rpc error: code = NotFound desc = could not find container \"f50edf569e1c2a0f9e3f3376c29bb5374e252c00ae6839551dc0ced0a59bfe16\": container with ID starting with f50edf569e1c2a0f9e3f3376c29bb5374e252c00ae6839551dc0ced0a59bfe16 not found: ID does not exist"
Oct 30 01:13:54.101867 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.101831    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"]
Oct 30 01:13:54.102638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.102611    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:13:54.102780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.102640    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:13:54.102780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.102658    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt9vs\" (UniqueName: \"kubernetes.io/projected/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-kube-api-access-bt9vs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:13:54.102780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.102671    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30fbfcf0-f18a-4949-92d8-fbbb726ed62d-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:13:54.106020 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.105996    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-62f5d1/auth-test-model-62f5d1-kserve-74557c65f7-6nzhb"]
Oct 30 01:13:54.264325 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.264297    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30fbfcf0-f18a-4949-92d8-fbbb726ed62d" path="/var/lib/kubelet/pods/30fbfcf0-f18a-4949-92d8-fbbb726ed62d/volumes"
Oct 30 01:13:54.593727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.593566    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"]
Oct 30 01:13:54.594660 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.594636    6099 cpu_manager.go:414] "containerMap: removing container" podUID="30fbfcf0-f18a-4949-92d8-fbbb726ed62d" containerName="main"
Oct 30 01:13:54.594660 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.594660    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30fbfcf0-f18a-4949-92d8-fbbb726ed62d" containerName="main"
Oct 30 01:13:54.594995 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.594978    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="30fbfcf0-f18a-4949-92d8-fbbb726ed62d" containerName="main"
Oct 30 01:13:54.598391 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.598370    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.623069 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.623039    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-60e8f4-openshift-default-dockercfg-665t8\""
Oct 30 01:13:54.624116 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.624085    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"]
Oct 30 01:13:54.713147 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.713110    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-podinfo\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.713294 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.713224    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6r8m\" (UniqueName: \"kubernetes.io/projected/b9004b35-ce7c-45b6-a0d0-94a16203ece2-kube-api-access-t6r8m\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.713294 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.713262    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-workload-certs\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.713387 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.713343    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istiod-ca-cert\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.713387 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.713365    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-workload-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.713491 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.713469    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-token\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.713564 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.713503    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-credential-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.713633 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.713622    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-data\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.713673 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.713659    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-envoy\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.789730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.788795    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Oct 30 01:13:54.789730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.788857    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz" podUID="e2c83697-c779-4359-804d-b6b4b4d5e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Oct 30 01:13:54.816788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.815427    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6r8m\" (UniqueName: \"kubernetes.io/projected/b9004b35-ce7c-45b6-a0d0-94a16203ece2-kube-api-access-t6r8m\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.816788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.815487    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-workload-certs\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.816788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.815652    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istiod-ca-cert\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.816788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.815690    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-workload-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.816788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.815880    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-token\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.816788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.815970    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-credential-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.816788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.816188    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istiod-ca-cert\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.816788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.816200    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-data\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.816788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.816263    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-envoy\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.816788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.816360    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-podinfo\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.816788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.816408    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-data\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.816788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.816484    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-workload-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.816788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.816554    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-credential-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.818876 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.816883    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-workload-certs\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.818954 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.818878    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-envoy\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.819043 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.819019    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-podinfo\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.839352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.839319    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-token\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.839944 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.839928    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6r8m\" (UniqueName: \"kubernetes.io/projected/b9004b35-ce7c-45b6-a0d0-94a16203ece2-kube-api-access-t6r8m\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:54.913481 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:54.913397    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:55.264362 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:13:55.264333    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9004b35_ce7c_45b6_a0d0_94a16203ece2.slice/crio-07b8b408b4b6d9d29049a37cd1650b94175263c8cb38f4eccf076ca13b852182 WatchSource:0}: Error finding container 07b8b408b4b6d9d29049a37cd1650b94175263c8cb38f4eccf076ca13b852182: Status 404 returned error can't find the container with id 07b8b408b4b6d9d29049a37cd1650b94175263c8cb38f4eccf076ca13b852182
Oct 30 01:13:55.264555 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:55.264532    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"]
Oct 30 01:13:55.284139 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:55.268166    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:13:55.284139 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:55.268251    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:13:55.284139 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:55.268296    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:13:55.780058 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:55.780023    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Oct 30 01:13:55.780781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:55.780750    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz" podUID="e2c83697-c779-4359-804d-b6b4b4d5e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Oct 30 01:13:56.054838 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:56.054804    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc" event={"ID":"b9004b35-ce7c-45b6-a0d0-94a16203ece2","Type":"ContainerStarted","Data":"d4304c788ceea067ebdfd5204554a2a9388b6cfab8db76af130bbecb00c45b69"}
Oct 30 01:13:56.055020 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:56.054857    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc" event={"ID":"b9004b35-ce7c-45b6-a0d0-94a16203ece2","Type":"ContainerStarted","Data":"07b8b408b4b6d9d29049a37cd1650b94175263c8cb38f4eccf076ca13b852182"}
Oct 30 01:13:56.161281 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:56.161218    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc" podStartSLOduration=2.161200015 podStartE2EDuration="2.161200015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:56.147843392 +0000 UTC m=+2330.004751176" watchObservedRunningTime="2026-10-30 01:13:56.161200015 +0000 UTC m=+2330.018107770"
Oct 30 01:13:56.784042 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:56.784007    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:13:56.787937 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:56.787906    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:13:56.914805 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:56.914774    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:13:56.916155 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:56.916125    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body=
Oct 30 01:13:56.916301 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:56.916185    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc" podUID="b9004b35-ce7c-45b6-a0d0-94a16203ece2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused"
Oct 30 01:13:57.914271 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:57.914235    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body=
Oct 30 01:13:57.914819 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:57.914288    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc" podUID="b9004b35-ce7c-45b6-a0d0-94a16203ece2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused"
Oct 30 01:13:58.914769 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:58.914690    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body=
Oct 30 01:13:58.915237 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:58.914801    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc" podUID="b9004b35-ce7c-45b6-a0d0-94a16203ece2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused"
Oct 30 01:13:58.989876 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:58.989836    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j"
Oct 30 01:13:59.914362 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:59.914327    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body=
Oct 30 01:13:59.914563 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:13:59.914390    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc" podUID="b9004b35-ce7c-45b6-a0d0-94a16203ece2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused"
Oct 30 01:14:00.154919 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.154877    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc" podUID="8af4b8ef-f359-4e4b-8474-dda967065618" containerName="main" containerID="cri-o://96406d2129e7b1b4997c885c91499dcf119b7e2e0313d89a15b2b4f4a0c5f813" gracePeriod=45
Oct 30 01:14:00.631859 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.631830    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:14:00.755886 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.755852    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8af4b8ef-f359-4e4b-8474-dda967065618-kube-api-access-8d2m6\" (UniqueName: \"kubernetes.io/projected/8af4b8ef-f359-4e4b-8474-dda967065618-kube-api-access-8d2m6\") pod \"8af4b8ef-f359-4e4b-8474-dda967065618\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") "
Oct 30 01:14:00.756089 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.755979    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-dshm\") pod \"8af4b8ef-f359-4e4b-8474-dda967065618\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") "
Oct 30 01:14:00.756089 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.756079    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-home\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-home\") pod \"8af4b8ef-f359-4e4b-8474-dda967065618\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") "
Oct 30 01:14:00.756252 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.756232    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8af4b8ef-f359-4e4b-8474-dda967065618-tls-certs\" (UniqueName: \"kubernetes.io/secret/8af4b8ef-f359-4e4b-8474-dda967065618-tls-certs\") pod \"8af4b8ef-f359-4e4b-8474-dda967065618\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") "
Oct 30 01:14:00.756326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.756307    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-model-cache\") pod \"8af4b8ef-f359-4e4b-8474-dda967065618\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") "
Oct 30 01:14:00.756390 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.756367    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-home" pod "8af4b8ef-f359-4e4b-8474-dda967065618" (UID: "8af4b8ef-f359-4e4b-8474-dda967065618"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:00.756452 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.756424    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-tmp-dir\") pod \"8af4b8ef-f359-4e4b-8474-dda967065618\" (UID: \"8af4b8ef-f359-4e4b-8474-dda967065618\") "
Oct 30 01:14:00.756961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.756631    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-model-cache" pod "8af4b8ef-f359-4e4b-8474-dda967065618" (UID: "8af4b8ef-f359-4e4b-8474-dda967065618"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:00.756961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.756901    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-tmp-dir" pod "8af4b8ef-f359-4e4b-8474-dda967065618" (UID: "8af4b8ef-f359-4e4b-8474-dda967065618"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:00.757299 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.757278    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:00.757299 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.757299    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:00.757481 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.757309    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:00.758269 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.758246    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af4b8ef-f359-4e4b-8474-dda967065618-kube-api-access-8d2m6" pod "8af4b8ef-f359-4e4b-8474-dda967065618" (UID: "8af4b8ef-f359-4e4b-8474-dda967065618"). InnerVolumeSpecName "kube-api-access-8d2m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:00.758568 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.758549    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-dshm" pod "8af4b8ef-f359-4e4b-8474-dda967065618" (UID: "8af4b8ef-f359-4e4b-8474-dda967065618"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:00.758870 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.758846    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8af4b8ef-f359-4e4b-8474-dda967065618-tls-certs" pod "8af4b8ef-f359-4e4b-8474-dda967065618" (UID: "8af4b8ef-f359-4e4b-8474-dda967065618"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:00.859095 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.859014    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d2m6\" (UniqueName: \"kubernetes.io/projected/8af4b8ef-f359-4e4b-8474-dda967065618-kube-api-access-8d2m6\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:00.859095 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.859045    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8af4b8ef-f359-4e4b-8474-dda967065618-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:00.859095 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.859059    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8af4b8ef-f359-4e4b-8474-dda967065618-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:00.914452 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.914419    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body=
Oct 30 01:14:00.914678 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:00.914482    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc" podUID="b9004b35-ce7c-45b6-a0d0-94a16203ece2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused"
Oct 30 01:14:01.127008 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:01.126915    6099 generic.go:356] "Generic (PLEG): container finished" podID="8af4b8ef-f359-4e4b-8474-dda967065618" containerID="96406d2129e7b1b4997c885c91499dcf119b7e2e0313d89a15b2b4f4a0c5f813" exitCode=0
Oct 30 01:14:01.127008 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:01.126966    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"
Oct 30 01:14:01.127008 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:01.126997    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc" event={"ID":"8af4b8ef-f359-4e4b-8474-dda967065618","Type":"ContainerDied","Data":"96406d2129e7b1b4997c885c91499dcf119b7e2e0313d89a15b2b4f4a0c5f813"}
Oct 30 01:14:01.127305 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:01.127050    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc" event={"ID":"8af4b8ef-f359-4e4b-8474-dda967065618","Type":"ContainerDied","Data":"30c755e0bd4c36cf8ed620dc6be88f7148372f600f7b111e79191e2b2c1c67a6"}
Oct 30 01:14:01.127305 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:01.127080    6099 scope.go:122] "RemoveContainer" containerID="96406d2129e7b1b4997c885c91499dcf119b7e2e0313d89a15b2b4f4a0c5f813"
Oct 30 01:14:01.143804 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:01.143785    6099 scope.go:122] "RemoveContainer" containerID="96406d2129e7b1b4997c885c91499dcf119b7e2e0313d89a15b2b4f4a0c5f813"
Oct 30 01:14:01.144344 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:14:01.144322    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96406d2129e7b1b4997c885c91499dcf119b7e2e0313d89a15b2b4f4a0c5f813\": container with ID starting with 96406d2129e7b1b4997c885c91499dcf119b7e2e0313d89a15b2b4f4a0c5f813 not found: ID does not exist" containerID="96406d2129e7b1b4997c885c91499dcf119b7e2e0313d89a15b2b4f4a0c5f813"
Oct 30 01:14:01.144437 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:01.144351    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96406d2129e7b1b4997c885c91499dcf119b7e2e0313d89a15b2b4f4a0c5f813"} err="failed to get container status \"96406d2129e7b1b4997c885c91499dcf119b7e2e0313d89a15b2b4f4a0c5f813\": rpc error: code = NotFound desc = could not find container \"96406d2129e7b1b4997c885c91499dcf119b7e2e0313d89a15b2b4f4a0c5f813\": container with ID starting with 96406d2129e7b1b4997c885c91499dcf119b7e2e0313d89a15b2b4f4a0c5f813 not found: ID does not exist"
Oct 30 01:14:01.197191 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:01.197156    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"]
Oct 30 01:14:01.203555 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:01.203526    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-969b92/rate-test-model-969b92-kserve-5764b78489-r4vhc"]
Oct 30 01:14:02.076555 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:02.076526    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:14:02.076988 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:02.076938    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:14:02.077765 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:02.077738    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:14:02.264362 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:02.264329    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8af4b8ef-f359-4e4b-8474-dda967065618" path="/var/lib/kubelet/pods/8af4b8ef-f359-4e4b-8474-dda967065618/volumes"
Oct 30 01:14:03.038417 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:03.038379    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd"]
Oct 30 01:14:03.053390 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:03.053355    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d269878c-z8qpd"]
Oct 30 01:14:04.264203 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:04.264169    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40960901-5a4f-4584-9a78-fe0f4a8eae0a" path="/var/lib/kubelet/pods/40960901-5a4f-4584-9a78-fe0f4a8eae0a/volumes"
Oct 30 01:14:12.206964 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.206923    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"]
Oct 30 01:14:12.208171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.208144    6099 cpu_manager.go:414] "containerMap: removing container" podUID="8af4b8ef-f359-4e4b-8474-dda967065618" containerName="main"
Oct 30 01:14:12.208171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.208170    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8af4b8ef-f359-4e4b-8474-dda967065618" containerName="main"
Oct 30 01:14:12.208375 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.208363    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="8af4b8ef-f359-4e4b-8474-dda967065618" containerName="main"
Oct 30 01:14:12.212744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.212676    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.240767 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.240536    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-8e24a5\"/\"kube-root-ca.crt\""
Oct 30 01:14:12.240767 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.240595    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-8e24a5\"/\"openshift-service-ca.crt\""
Oct 30 01:14:12.240767 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.240748    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-8e24a5\"/\"default-dockercfg-k7knd\""
Oct 30 01:14:12.241807 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.240860    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-8e24a5\"/\"auth-test-model-8e24a5-kserve-self-signed-certs\""
Oct 30 01:14:12.243794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.243770    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"]
Oct 30 01:14:12.358532 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.358494    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-model-cache\") pod \"auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") " pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.358751 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.358645    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrp5b\" (UniqueName: \"kubernetes.io/projected/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-kube-api-access-vrp5b\") pod \"auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") " pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.358818 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.358780    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-tls-certs\") pod \"auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") " pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.358899 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.358883    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-home\") pod \"auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") " pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.358950 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.358909    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-dshm\") pod \"auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") " pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.359089 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.359066    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-tmp-dir\") pod \"auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") " pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.483628 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.480946    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrp5b\" (UniqueName: \"kubernetes.io/projected/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-kube-api-access-vrp5b\") pod \"auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") " pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.483628 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.481113    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-tls-certs\") pod \"auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") " pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.483628 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.481239    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-home\") pod \"auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") " pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.483628 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.481266    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-dshm\") pod \"auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") " pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.483628 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.481325    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-tmp-dir\") pod \"auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") " pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.483628 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.481512    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-model-cache\") pod \"auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") " pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.483628 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.482065    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-model-cache\") pod \"auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") " pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.489119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.487057    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-tmp-dir\") pod \"auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") " pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.489119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.487350    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-home\") pod \"auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") " pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.503618 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.503578    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-tls-certs\") pod \"auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") " pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.508399 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.508369    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrp5b\" (UniqueName: \"kubernetes.io/projected/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-kube-api-access-vrp5b\") pod \"auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") " pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.512740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.512040    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-dshm\") pod \"auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") " pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.553066 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.553030    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:12.900700 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:14:12.900662    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e96d7ed_1b43_45d9_9d84_64b9aaf7e694.slice/crio-410f4c49163e2b93964692db08039aa8045a8eb40a210187f0c90722ce86be30 WatchSource:0}: Error finding container 410f4c49163e2b93964692db08039aa8045a8eb40a210187f0c90722ce86be30: Status 404 returned error can't find the container with id 410f4c49163e2b93964692db08039aa8045a8eb40a210187f0c90722ce86be30
Oct 30 01:14:12.901024 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:12.901001    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"]
Oct 30 01:14:13.033733 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:13.033689    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf"]
Oct 30 01:14:13.037678 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:13.037650    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85089ec9-sbkgf"]
Oct 30 01:14:13.253528 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:13.253442    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2" event={"ID":"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694","Type":"ContainerStarted","Data":"6ca883d51f797409e57035d6721e180878a99b5bd1b90e106acf32ca954cb302"}
Oct 30 01:14:13.253528 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:13.253478    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2" event={"ID":"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694","Type":"ContainerStarted","Data":"410f4c49163e2b93964692db08039aa8045a8eb40a210187f0c90722ce86be30"}
Oct 30 01:14:13.296875 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:13.296824    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2" podStartSLOduration=1.296811531 podStartE2EDuration="1.296811531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:13.295409706 +0000 UTC m=+2347.152317461" watchObservedRunningTime="2026-10-30 01:14:13.296811531 +0000 UTC m=+2347.153719287"
Oct 30 01:14:13.350539 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:13.350508    6099 scope.go:122] "RemoveContainer" containerID="055b0b5255a899c0f6ebd4b7b55e2c75ba4d602628f6ff817dd02e4a2b4cc4af"
Oct 30 01:14:13.362486 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:13.362466    6099 scope.go:122] "RemoveContainer" containerID="8e6154796d5712c38171030455b6cccf6e503ffd38e689cec46873a484059df1"
Oct 30 01:14:13.374371 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:13.374345    6099 scope.go:122] "RemoveContainer" containerID="450dcd6b2434b2ebe33281cbfab2e1560406f1c73079a4f859015e6f9aad7971"
Oct 30 01:14:13.384910 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:13.384887    6099 scope.go:122] "RemoveContainer" containerID="dfc021b2f376d2310a547237a431723cfd022efd87812c725fe9b476e1b149aa"
Oct 30 01:14:13.555075 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:13.555050    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:14.052628 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:14.052598    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"]
Oct 30 01:14:14.295362 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:14.295327    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f855a43-e9d5-4e14-9142-3a39084f2098" path="/var/lib/kubelet/pods/2f855a43-e9d5-4e14-9142-3a39084f2098/volumes"
Oct 30 01:14:15.193162 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.193130    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"]
Oct 30 01:14:15.199414 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.199387    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.227728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.226434    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-481249-openshift-default-dockercfg-g5l4g\""
Oct 30 01:14:15.227728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.226496    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"]
Oct 30 01:14:15.260512 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.258006    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-podinfo\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.260512 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.258072    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-envoy\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.260512 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.258168    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-token\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.260512 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.258198    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-credential-socket\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.260512 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.258308    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-workload-certs\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.260512 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.258361    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g66jc\" (UniqueName: \"kubernetes.io/projected/87d1ee54-2797-4936-a3a3-44624f7c15c8-kube-api-access-g66jc\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.260512 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.258554    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-data\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.260512 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.258594    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-workload-socket\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.260512 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.258624    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87d1ee54-2797-4936-a3a3-44624f7c15c8-istiod-ca-cert\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.367730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.361020    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-envoy\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.367730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.361160    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-token\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.367730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.361190    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-credential-socket\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.367730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.361314    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-workload-certs\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.367730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.361377    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g66jc\" (UniqueName: \"kubernetes.io/projected/87d1ee54-2797-4936-a3a3-44624f7c15c8-kube-api-access-g66jc\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.367730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.361567    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-data\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.367730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.361605    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-workload-socket\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.367730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.361634    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87d1ee54-2797-4936-a3a3-44624f7c15c8-istiod-ca-cert\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.367730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.361733    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-podinfo\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.367730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.365342    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-podinfo\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.406547 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.386578    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-credential-socket\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.548838 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.392106    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-workload-certs\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.556404 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.392635    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-data\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.556619 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.392892    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-workload-socket\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.556737 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.429471    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87d1ee54-2797-4936-a3a3-44624f7c15c8-istiod-ca-cert\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.556848 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.449608    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-envoy\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.570325 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.570294    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g66jc\" (UniqueName: \"kubernetes.io/projected/87d1ee54-2797-4936-a3a3-44624f7c15c8-kube-api-access-g66jc\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.578623 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.574026    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-token\") pod \"e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") " pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:15.814596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:15.814517    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:16.168998 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:16.168956    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"]
Oct 30 01:14:16.169094 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:14:16.168949    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d1ee54_2797_4936_a3a3_44624f7c15c8.slice/crio-b8e363766b89083a1325120df0e579b5317b70068546155a987f24ea7cf0f708 WatchSource:0}: Error finding container b8e363766b89083a1325120df0e579b5317b70068546155a987f24ea7cf0f708: Status 404 returned error can't find the container with id b8e363766b89083a1325120df0e579b5317b70068546155a987f24ea7cf0f708
Oct 30 01:14:16.172020 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:16.171971    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:14:16.172132 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:16.172063    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:14:16.172132 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:16.172109    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:14:16.589522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:16.589456    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9" event={"ID":"87d1ee54-2797-4936-a3a3-44624f7c15c8","Type":"ContainerStarted","Data":"3bf85cbe38b97a8dec59f3de6329addc6f202683d89930fd17bdf53370a74835"}
Oct 30 01:14:16.589522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:16.589519    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9" event={"ID":"87d1ee54-2797-4936-a3a3-44624f7c15c8","Type":"ContainerStarted","Data":"b8e363766b89083a1325120df0e579b5317b70068546155a987f24ea7cf0f708"}
Oct 30 01:14:16.635001 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:14:16.634960    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:14:16.636458 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:16.636416    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9" podStartSLOduration=1.6364021530000001 podStartE2EDuration="1.636402153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:16.634580763 +0000 UTC m=+2350.491488532" watchObservedRunningTime="2026-10-30 01:14:16.636402153 +0000 UTC m=+2350.493309907"
Oct 30 01:14:16.815239 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:16.815211    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:17.822180 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:17.822149    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:18.607389 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:18.607361    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:18.608250 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:18.608228    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:14:20.532213 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.532182    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"]
Oct 30 01:14:20.538178 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.538154    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.564567 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.564539    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"]
Oct 30 01:14:20.572538 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.571438    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-320045\"/\"default-dockercfg-57hsk\""
Oct 30 01:14:20.572538 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.571507    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-320045\"/\"kube-root-ca.crt\""
Oct 30 01:14:20.572538 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.571569    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-320045\"/\"auth-test-model-320045-kserve-self-signed-certs\""
Oct 30 01:14:20.572538 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.571661    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-320045\"/\"openshift-service-ca.crt\""
Oct 30 01:14:20.597437 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.597405    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt8mm\" (UniqueName: \"kubernetes.io/projected/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-kube-api-access-tt8mm\") pod \"auth-test-model-320045-kserve-54468b6b85-8zl26\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") " pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.597599 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.597575    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-tls-certs\") pod \"auth-test-model-320045-kserve-54468b6b85-8zl26\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") " pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.597674 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.597624    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-home\") pod \"auth-test-model-320045-kserve-54468b6b85-8zl26\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") " pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.597876 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.597852    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-dshm\") pod \"auth-test-model-320045-kserve-54468b6b85-8zl26\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") " pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.598068 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.598041    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-model-cache\") pod \"auth-test-model-320045-kserve-54468b6b85-8zl26\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") " pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.598238 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.598220    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-tmp-dir\") pod \"auth-test-model-320045-kserve-54468b6b85-8zl26\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") " pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.700063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.700027    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-model-cache\") pod \"auth-test-model-320045-kserve-54468b6b85-8zl26\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") " pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.700224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.700171    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-tmp-dir\") pod \"auth-test-model-320045-kserve-54468b6b85-8zl26\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") " pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.700300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.700254    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt8mm\" (UniqueName: \"kubernetes.io/projected/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-kube-api-access-tt8mm\") pod \"auth-test-model-320045-kserve-54468b6b85-8zl26\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") " pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.700363 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.700345    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-tls-certs\") pod \"auth-test-model-320045-kserve-54468b6b85-8zl26\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") " pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.700432 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.700383    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-home\") pod \"auth-test-model-320045-kserve-54468b6b85-8zl26\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") " pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.700495 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.700477    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-model-cache\") pod \"auth-test-model-320045-kserve-54468b6b85-8zl26\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") " pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.700554 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.700517    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-dshm\") pod \"auth-test-model-320045-kserve-54468b6b85-8zl26\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") " pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.701301 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.700679    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-home\") pod \"auth-test-model-320045-kserve-54468b6b85-8zl26\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") " pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.701301 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.701062    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-tmp-dir\") pod \"auth-test-model-320045-kserve-54468b6b85-8zl26\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") " pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.703306 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.703280    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-dshm\") pod \"auth-test-model-320045-kserve-54468b6b85-8zl26\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") " pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.703415 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.703313    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-tls-certs\") pod \"auth-test-model-320045-kserve-54468b6b85-8zl26\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") " pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.724090 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.724065    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt8mm\" (UniqueName: \"kubernetes.io/projected/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-kube-api-access-tt8mm\") pod \"auth-test-model-320045-kserve-54468b6b85-8zl26\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") " pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:20.890164 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:20.889434    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:21.467582 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:14:21.467548    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b163830_1b90_4bcc_8238_a08bc5b9cb7e.slice/crio-46a52887557b7a51c568cff4c9c1a6424dc182ea3c4837659bb1bc0cc2c161e8 WatchSource:0}: Error finding container 46a52887557b7a51c568cff4c9c1a6424dc182ea3c4837659bb1bc0cc2c161e8: Status 404 returned error can't find the container with id 46a52887557b7a51c568cff4c9c1a6424dc182ea3c4837659bb1bc0cc2c161e8
Oct 30 01:14:21.469680 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:21.469604    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"]
Oct 30 01:14:21.635792 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:21.635753    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26" event={"ID":"6b163830-1b90-4bcc-8238-a08bc5b9cb7e","Type":"ContainerStarted","Data":"ed2ee8dede2dbbd090295aad1b3c7d70ec78c3e6891de2866733491a260519c2"}
Oct 30 01:14:21.636164 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:21.635800    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26" event={"ID":"6b163830-1b90-4bcc-8238-a08bc5b9cb7e","Type":"ContainerStarted","Data":"46a52887557b7a51c568cff4c9c1a6424dc182ea3c4837659bb1bc0cc2c161e8"}
Oct 30 01:14:21.678608 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:21.678561    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26" podStartSLOduration=1.6785468799999999 podStartE2EDuration="1.67854688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:21.676664678 +0000 UTC m=+2355.533572656" watchObservedRunningTime="2026-10-30 01:14:21.67854688 +0000 UTC m=+2355.535454632"
Oct 30 01:14:21.905423 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:21.905394    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:22.403000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:22.402961    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:14:22.554655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:22.554620    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:22.569458 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:22.569417    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:22.658387 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:22.658305    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:14:29.080856 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.080812    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl" podUID="657a9d22-491f-4ef1-909b-a27511860627" containerName="main" containerID="cri-o://0512c0c70d00efd83c33301fb73d737f9d170b74629dafa00fbe9cef882c0b99" gracePeriod=45
Oct 30 01:14:29.214890 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.214857    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl" podUID="657a9d22-491f-4ef1-909b-a27511860627" containerName="main" probeResult="failure" output="Get \"https://10.134.1.0:8000/ready\": dial tcp 10.134.1.0:8000: connect: connection refused"
Oct 30 01:14:29.396492 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.396466    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:14:29.552085 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.552052    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657a9d22-491f-4ef1-909b-a27511860627-kube-api-access-xz6t5\" (UniqueName: \"kubernetes.io/projected/657a9d22-491f-4ef1-909b-a27511860627-kube-api-access-xz6t5\") pod \"657a9d22-491f-4ef1-909b-a27511860627\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") "
Oct 30 01:14:29.552365 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.552347    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-home\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-home\") pod \"657a9d22-491f-4ef1-909b-a27511860627\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") "
Oct 30 01:14:29.552578 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.552556    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/657a9d22-491f-4ef1-909b-a27511860627-tls-certs\" (UniqueName: \"kubernetes.io/secret/657a9d22-491f-4ef1-909b-a27511860627-tls-certs\") pod \"657a9d22-491f-4ef1-909b-a27511860627\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") "
Oct 30 01:14:29.552728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.552590    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-home" pod "657a9d22-491f-4ef1-909b-a27511860627" (UID: "657a9d22-491f-4ef1-909b-a27511860627"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:29.552728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.552610    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-tmp-dir\") pod \"657a9d22-491f-4ef1-909b-a27511860627\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") "
Oct 30 01:14:29.552728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.552658    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-model-cache\") pod \"657a9d22-491f-4ef1-909b-a27511860627\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") "
Oct 30 01:14:29.552728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.552716    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-dshm\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-dshm\") pod \"657a9d22-491f-4ef1-909b-a27511860627\" (UID: \"657a9d22-491f-4ef1-909b-a27511860627\") "
Oct 30 01:14:29.553025 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.552894    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-tmp-dir" pod "657a9d22-491f-4ef1-909b-a27511860627" (UID: "657a9d22-491f-4ef1-909b-a27511860627"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:29.553025 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.552911    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-model-cache" pod "657a9d22-491f-4ef1-909b-a27511860627" (UID: "657a9d22-491f-4ef1-909b-a27511860627"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:29.553620 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.553595    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:29.553733 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.553625    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:29.553733 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.553642    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:29.554638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.554611    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657a9d22-491f-4ef1-909b-a27511860627-kube-api-access-xz6t5" pod "657a9d22-491f-4ef1-909b-a27511860627" (UID: "657a9d22-491f-4ef1-909b-a27511860627"). InnerVolumeSpecName "kube-api-access-xz6t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:29.554784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.554755    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/657a9d22-491f-4ef1-909b-a27511860627-tls-certs" pod "657a9d22-491f-4ef1-909b-a27511860627" (UID: "657a9d22-491f-4ef1-909b-a27511860627"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:29.554874 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.554856    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-dshm" pod "657a9d22-491f-4ef1-909b-a27511860627" (UID: "657a9d22-491f-4ef1-909b-a27511860627"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:29.655614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.655543    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657a9d22-491f-4ef1-909b-a27511860627-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:29.655614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.655569    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657a9d22-491f-4ef1-909b-a27511860627-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:29.655614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:29.655578    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz6t5\" (UniqueName: \"kubernetes.io/projected/657a9d22-491f-4ef1-909b-a27511860627-kube-api-access-xz6t5\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:30.033103 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:30.033019    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76"]
Oct 30 01:14:30.046304 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:30.046270    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ba0956-vmq76"]
Oct 30 01:14:30.049391 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:30.049364    6099 generic.go:356] "Generic (PLEG): container finished" podID="657a9d22-491f-4ef1-909b-a27511860627" containerID="0512c0c70d00efd83c33301fb73d737f9d170b74629dafa00fbe9cef882c0b99" exitCode=0
Oct 30 01:14:30.049514 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:30.049417    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"
Oct 30 01:14:30.049514 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:30.049444    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl" event={"ID":"657a9d22-491f-4ef1-909b-a27511860627","Type":"ContainerDied","Data":"0512c0c70d00efd83c33301fb73d737f9d170b74629dafa00fbe9cef882c0b99"}
Oct 30 01:14:30.049514 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:30.049482    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl" event={"ID":"657a9d22-491f-4ef1-909b-a27511860627","Type":"ContainerDied","Data":"e76c9d6cb5fa813163696aa22e5382b69d484bceedc24f00bfbb0da805ac1f44"}
Oct 30 01:14:30.049514 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:30.049500    6099 scope.go:122] "RemoveContainer" containerID="0512c0c70d00efd83c33301fb73d737f9d170b74629dafa00fbe9cef882c0b99"
Oct 30 01:14:30.062334 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:30.062317    6099 scope.go:122] "RemoveContainer" containerID="0512c0c70d00efd83c33301fb73d737f9d170b74629dafa00fbe9cef882c0b99"
Oct 30 01:14:30.062622 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:14:30.062597    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0512c0c70d00efd83c33301fb73d737f9d170b74629dafa00fbe9cef882c0b99\": container with ID starting with 0512c0c70d00efd83c33301fb73d737f9d170b74629dafa00fbe9cef882c0b99 not found: ID does not exist" containerID="0512c0c70d00efd83c33301fb73d737f9d170b74629dafa00fbe9cef882c0b99"
Oct 30 01:14:30.062698 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:30.062633    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0512c0c70d00efd83c33301fb73d737f9d170b74629dafa00fbe9cef882c0b99"} err="failed to get container status \"0512c0c70d00efd83c33301fb73d737f9d170b74629dafa00fbe9cef882c0b99\": rpc error: code = NotFound desc = could not find container \"0512c0c70d00efd83c33301fb73d737f9d170b74629dafa00fbe9cef882c0b99\": container with ID starting with 0512c0c70d00efd83c33301fb73d737f9d170b74629dafa00fbe9cef882c0b99 not found: ID does not exist"
Oct 30 01:14:30.102801 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:30.102772    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"]
Oct 30 01:14:30.106378 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:30.106349    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ae06fe/rate-test-model-ae06fe-kserve-6c44c678df-pgswl"]
Oct 30 01:14:30.264696 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:30.264669    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="657a9d22-491f-4ef1-909b-a27511860627" path="/var/lib/kubelet/pods/657a9d22-491f-4ef1-909b-a27511860627/volumes"
Oct 30 01:14:30.265252 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:30.265235    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0638e9c-11e9-4aa5-b98d-15b8760b6399" path="/var/lib/kubelet/pods/e0638e9c-11e9-4aa5-b98d-15b8760b6399/volumes"
Oct 30 01:14:30.920330 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:30.920299    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:30.939418 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:30.939073    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:31.073074 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:31.073046    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:14:40.586671 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.586556    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"]
Oct 30 01:14:40.587863 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.587842    6099 cpu_manager.go:414] "containerMap: removing container" podUID="657a9d22-491f-4ef1-909b-a27511860627" containerName="main"
Oct 30 01:14:40.587955 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.587865    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657a9d22-491f-4ef1-909b-a27511860627" containerName="main"
Oct 30 01:14:40.588138 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.588124    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="657a9d22-491f-4ef1-909b-a27511860627" containerName="main"
Oct 30 01:14:40.591297 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.591278    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.606372 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.606344    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 01:14:40.606488 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.606383    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 01:14:40.606488 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.606418    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 01:14:40.606571 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.606544    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pqwzr\""
Oct 30 01:14:40.622871 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.622846    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"]
Oct 30 01:14:40.744443 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.744411    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.744627 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.744479    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srsf6\" (UniqueName: \"kubernetes.io/projected/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-kube-api-access-srsf6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.744627 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.744580    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.744762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.744742    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.744809 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.744778    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.744869 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.744857    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.744931 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.744919    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.846686 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.846605    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.846686 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.846647    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.846686 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.846683    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.847006 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.846787    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.847006 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.846987    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.847130 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.847073    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srsf6\" (UniqueName: \"kubernetes.io/projected/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-kube-api-access-srsf6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.847189 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.847151    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.847629 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.847601    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.849308 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.849283    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.849555 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.849533    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.849877 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.849855    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.850180 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.850161    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.869615 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.869592    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.869955 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.869936    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srsf6\" (UniqueName: \"kubernetes.io/projected/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-kube-api-access-srsf6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvbtj\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:40.903563 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:40.903540    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:41.232538 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:14:41.232005    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2a8c69_5075_4f11_9920_2aeb2ee26ba8.slice/crio-0b50483b2480b76cc9fa62d3a6bbd8bc6a8233a7716ddd4db83c5f81e10e13c5 WatchSource:0}: Error finding container 0b50483b2480b76cc9fa62d3a6bbd8bc6a8233a7716ddd4db83c5f81e10e13c5: Status 404 returned error can't find the container with id 0b50483b2480b76cc9fa62d3a6bbd8bc6a8233a7716ddd4db83c5f81e10e13c5
Oct 30 01:14:41.232922 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:41.232894    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"]
Oct 30 01:14:41.236144 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:41.236083    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:14:41.236267 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:41.236172    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:14:41.774411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:41.774373    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"]
Oct 30 01:14:42.200249 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:42.184890    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj" event={"ID":"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8","Type":"ContainerStarted","Data":"97c5ff2a1ca4c81eb8207a3636f412e526cbe2ae0fe0397c51c54be073c45c09"}
Oct 30 01:14:42.200249 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:42.185006    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj" event={"ID":"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8","Type":"ContainerStarted","Data":"0b50483b2480b76cc9fa62d3a6bbd8bc6a8233a7716ddd4db83c5f81e10e13c5"}
Oct 30 01:14:42.200249 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:42.186442    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:42.200249 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:42.188203    6099 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wvbtj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:14:42.200249 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:42.188246    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj" podUID="8c2a8c69-5075-4f11-9920-2aeb2ee26ba8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:14:42.235524 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:42.235458    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj" podStartSLOduration=2.23543937 podStartE2EDuration="2.23543937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:42.231286844 +0000 UTC m=+2376.088194600" watchObservedRunningTime="2026-10-30 01:14:42.23543937 +0000 UTC m=+2376.092347129"
Oct 30 01:14:42.286729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:42.286652    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"]
Oct 30 01:14:43.197784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:43.197746    6099 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wvbtj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:14:43.198313 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:43.197819    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj" podUID="8c2a8c69-5075-4f11-9920-2aeb2ee26ba8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:14:44.205845 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:44.205814    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:14:46.631451 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:14:46.631415    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:14:50.371488 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:50.371450    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"]
Oct 30 01:14:56.813375 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:56.813333    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk" podUID="5de95229-e8f1-483d-8540-15e1d33182ea" containerName="main" containerID="cri-o://7e4ae806feb805b6690342d78d76d18f9bf289eea700aef79f666a04c799069c" gracePeriod=45
Oct 30 01:14:57.132782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.132612    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:14:57.256599 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.256565    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-dshm\") pod \"5de95229-e8f1-483d-8540-15e1d33182ea\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") "
Oct 30 01:14:57.256821 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.256656    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-home\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-home\") pod \"5de95229-e8f1-483d-8540-15e1d33182ea\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") "
Oct 30 01:14:57.256821 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.256747    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-model-cache\") pod \"5de95229-e8f1-483d-8540-15e1d33182ea\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") "
Oct 30 01:14:57.256821 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.256809    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-tmp-dir\") pod \"5de95229-e8f1-483d-8540-15e1d33182ea\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") "
Oct 30 01:14:57.257001 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.256911    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-home" pod "5de95229-e8f1-483d-8540-15e1d33182ea" (UID: "5de95229-e8f1-483d-8540-15e1d33182ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:57.257064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.256979    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5de95229-e8f1-483d-8540-15e1d33182ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/5de95229-e8f1-483d-8540-15e1d33182ea-tls-certs\") pod \"5de95229-e8f1-483d-8540-15e1d33182ea\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") "
Oct 30 01:14:57.257119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.257051    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-model-cache" pod "5de95229-e8f1-483d-8540-15e1d33182ea" (UID: "5de95229-e8f1-483d-8540-15e1d33182ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:57.257119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.257082    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de95229-e8f1-483d-8540-15e1d33182ea-kube-api-access-v4p2s\" (UniqueName: \"kubernetes.io/projected/5de95229-e8f1-483d-8540-15e1d33182ea-kube-api-access-v4p2s\") pod \"5de95229-e8f1-483d-8540-15e1d33182ea\" (UID: \"5de95229-e8f1-483d-8540-15e1d33182ea\") "
Oct 30 01:14:57.257208 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.257122    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-tmp-dir" pod "5de95229-e8f1-483d-8540-15e1d33182ea" (UID: "5de95229-e8f1-483d-8540-15e1d33182ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:57.257825 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.257802    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:57.257956 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.257825    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:57.257956 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.257856    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:57.258922 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.258898    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-dshm" pod "5de95229-e8f1-483d-8540-15e1d33182ea" (UID: "5de95229-e8f1-483d-8540-15e1d33182ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:57.259589 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.259562    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de95229-e8f1-483d-8540-15e1d33182ea-kube-api-access-v4p2s" pod "5de95229-e8f1-483d-8540-15e1d33182ea" (UID: "5de95229-e8f1-483d-8540-15e1d33182ea"). InnerVolumeSpecName "kube-api-access-v4p2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:57.259683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.259657    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5de95229-e8f1-483d-8540-15e1d33182ea-tls-certs" pod "5de95229-e8f1-483d-8540-15e1d33182ea" (UID: "5de95229-e8f1-483d-8540-15e1d33182ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:57.340435 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.340366    6099 generic.go:356] "Generic (PLEG): container finished" podID="5de95229-e8f1-483d-8540-15e1d33182ea" containerID="7e4ae806feb805b6690342d78d76d18f9bf289eea700aef79f666a04c799069c" exitCode=0
Oct 30 01:14:57.340435 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.340407    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"
Oct 30 01:14:57.340621 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.340444    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk" event={"ID":"5de95229-e8f1-483d-8540-15e1d33182ea","Type":"ContainerDied","Data":"7e4ae806feb805b6690342d78d76d18f9bf289eea700aef79f666a04c799069c"}
Oct 30 01:14:57.340621 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.340489    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk" event={"ID":"5de95229-e8f1-483d-8540-15e1d33182ea","Type":"ContainerDied","Data":"1bbd12531bed2a5babdebc6e90a13ea499c21f6feb37739718cdc72b6bd5c747"}
Oct 30 01:14:57.340621 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.340509    6099 scope.go:122] "RemoveContainer" containerID="7e4ae806feb805b6690342d78d76d18f9bf289eea700aef79f666a04c799069c"
Oct 30 01:14:57.351078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.351046    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf" podUID="5f0f160f-9861-48e1-8a8d-6e33f9a46a46" containerName="main" containerID="cri-o://4fd468433f433cb2dd2aa01f723359425c5cbc596275ac168dd98061a6dddd2b" gracePeriod=45
Oct 30 01:14:57.353356 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.353334    6099 scope.go:122] "RemoveContainer" containerID="7e4ae806feb805b6690342d78d76d18f9bf289eea700aef79f666a04c799069c"
Oct 30 01:14:57.353825 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:14:57.353804    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e4ae806feb805b6690342d78d76d18f9bf289eea700aef79f666a04c799069c\": container with ID starting with 7e4ae806feb805b6690342d78d76d18f9bf289eea700aef79f666a04c799069c not found: ID does not exist" containerID="7e4ae806feb805b6690342d78d76d18f9bf289eea700aef79f666a04c799069c"
Oct 30 01:14:57.353915 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.353831    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4ae806feb805b6690342d78d76d18f9bf289eea700aef79f666a04c799069c"} err="failed to get container status \"7e4ae806feb805b6690342d78d76d18f9bf289eea700aef79f666a04c799069c\": rpc error: code = NotFound desc = could not find container \"7e4ae806feb805b6690342d78d76d18f9bf289eea700aef79f666a04c799069c\": container with ID starting with 7e4ae806feb805b6690342d78d76d18f9bf289eea700aef79f666a04c799069c not found: ID does not exist"
Oct 30 01:14:57.359343 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.359315    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5de95229-e8f1-483d-8540-15e1d33182ea-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:57.359445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.359346    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de95229-e8f1-483d-8540-15e1d33182ea-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:57.359445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.359360    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4p2s\" (UniqueName: \"kubernetes.io/projected/5de95229-e8f1-483d-8540-15e1d33182ea-kube-api-access-v4p2s\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:57.396875 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.396851    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"]
Oct 30 01:14:57.400899 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.400879    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-ea2547/sub-test-model-ea2547-kserve-67f8f9d6df-sl9jk"]
Oct 30 01:14:57.657545 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.657521    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:14:57.766456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.766424    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-kube-api-access-jmr7z\" (UniqueName: \"kubernetes.io/projected/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-kube-api-access-jmr7z\") pod \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") "
Oct 30 01:14:57.766658 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.766610    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-model-cache\") pod \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") "
Oct 30 01:14:57.766821 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.766800    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-tmp-dir\") pod \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") "
Oct 30 01:14:57.766970 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.766938    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-model-cache" pod "5f0f160f-9861-48e1-8a8d-6e33f9a46a46" (UID: "5f0f160f-9861-48e1-8a8d-6e33f9a46a46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:57.767063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.767044    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-tmp-dir" pod "5f0f160f-9861-48e1-8a8d-6e33f9a46a46" (UID: "5f0f160f-9861-48e1-8a8d-6e33f9a46a46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:57.767125 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.767108    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-tls-certs\") pod \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") "
Oct 30 01:14:57.767224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.767209    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-home\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-home\") pod \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") "
Oct 30 01:14:57.767297 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.767282    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-dshm\") pod \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\" (UID: \"5f0f160f-9861-48e1-8a8d-6e33f9a46a46\") "
Oct 30 01:14:57.767515 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.767484    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-home" pod "5f0f160f-9861-48e1-8a8d-6e33f9a46a46" (UID: "5f0f160f-9861-48e1-8a8d-6e33f9a46a46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:57.768835 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.768793    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:57.768835 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.768821    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:57.768835 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.768832    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:57.770078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.770052    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-tls-certs" pod "5f0f160f-9861-48e1-8a8d-6e33f9a46a46" (UID: "5f0f160f-9861-48e1-8a8d-6e33f9a46a46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:57.770173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.770093    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-dshm" pod "5f0f160f-9861-48e1-8a8d-6e33f9a46a46" (UID: "5f0f160f-9861-48e1-8a8d-6e33f9a46a46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:57.770173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.770093    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-kube-api-access-jmr7z" pod "5f0f160f-9861-48e1-8a8d-6e33f9a46a46" (UID: "5f0f160f-9861-48e1-8a8d-6e33f9a46a46"). InnerVolumeSpecName "kube-api-access-jmr7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:57.872731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.870923    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmr7z\" (UniqueName: \"kubernetes.io/projected/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-kube-api-access-jmr7z\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:57.872731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.870959    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:57.872731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:57.870973    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f0f160f-9861-48e1-8a8d-6e33f9a46a46-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:14:58.264251 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:58.264217    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de95229-e8f1-483d-8540-15e1d33182ea" path="/var/lib/kubelet/pods/5de95229-e8f1-483d-8540-15e1d33182ea/volumes"
Oct 30 01:14:58.350779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:58.350740    6099 generic.go:356] "Generic (PLEG): container finished" podID="5f0f160f-9861-48e1-8a8d-6e33f9a46a46" containerID="4fd468433f433cb2dd2aa01f723359425c5cbc596275ac168dd98061a6dddd2b" exitCode=0
Oct 30 01:14:58.350958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:58.350784    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf" event={"ID":"5f0f160f-9861-48e1-8a8d-6e33f9a46a46","Type":"ContainerDied","Data":"4fd468433f433cb2dd2aa01f723359425c5cbc596275ac168dd98061a6dddd2b"}
Oct 30 01:14:58.350958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:58.350834    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf" event={"ID":"5f0f160f-9861-48e1-8a8d-6e33f9a46a46","Type":"ContainerDied","Data":"f6f45d2ab2939bd50d725cc05ca65a719a3f1aa66b3a74f00ad383b9f16383f1"}
Oct 30 01:14:58.350958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:58.350853    6099 scope.go:122] "RemoveContainer" containerID="4fd468433f433cb2dd2aa01f723359425c5cbc596275ac168dd98061a6dddd2b"
Oct 30 01:14:58.350958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:58.350793    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"
Oct 30 01:14:58.365800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:58.365779    6099 scope.go:122] "RemoveContainer" containerID="4fd468433f433cb2dd2aa01f723359425c5cbc596275ac168dd98061a6dddd2b"
Oct 30 01:14:58.366339 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:14:58.366317    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd468433f433cb2dd2aa01f723359425c5cbc596275ac168dd98061a6dddd2b\": container with ID starting with 4fd468433f433cb2dd2aa01f723359425c5cbc596275ac168dd98061a6dddd2b not found: ID does not exist" containerID="4fd468433f433cb2dd2aa01f723359425c5cbc596275ac168dd98061a6dddd2b"
Oct 30 01:14:58.366438 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:58.366343    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd468433f433cb2dd2aa01f723359425c5cbc596275ac168dd98061a6dddd2b"} err="failed to get container status \"4fd468433f433cb2dd2aa01f723359425c5cbc596275ac168dd98061a6dddd2b\": rpc error: code = NotFound desc = could not find container \"4fd468433f433cb2dd2aa01f723359425c5cbc596275ac168dd98061a6dddd2b\": container with ID starting with 4fd468433f433cb2dd2aa01f723359425c5cbc596275ac168dd98061a6dddd2b not found: ID does not exist"
Oct 30 01:14:58.403898 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:58.403869    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"]
Oct 30 01:14:58.415415 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:58.415390    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf"]
Oct 30 01:14:58.550022 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:14:58.549991    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-d01638/auth-test-model-d01638-kserve-559c6d4469-grrdf" podUID="5f0f160f-9861-48e1-8a8d-6e33f9a46a46" containerName="main" probeResult="failure" output="Get \"https://10.134.1.18:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:15:00.151908 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.151874    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw"]
Oct 30 01:15:00.152745 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.152729    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5de95229-e8f1-483d-8540-15e1d33182ea" containerName="main"
Oct 30 01:15:00.152806 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.152745    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de95229-e8f1-483d-8540-15e1d33182ea" containerName="main"
Oct 30 01:15:00.152806 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.152762    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5f0f160f-9861-48e1-8a8d-6e33f9a46a46" containerName="main"
Oct 30 01:15:00.152806 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.152767    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0f160f-9861-48e1-8a8d-6e33f9a46a46" containerName="main"
Oct 30 01:15:00.152907 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.152892    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de95229-e8f1-483d-8540-15e1d33182ea" containerName="main"
Oct 30 01:15:00.152907 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.152901    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0f160f-9861-48e1-8a8d-6e33f9a46a46" containerName="main"
Oct 30 01:15:00.158063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.158036    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw"
Oct 30 01:15:00.168542 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.168519    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr"]
Oct 30 01:15:00.174429 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.174406    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr"
Oct 30 01:15:00.176511 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.176441    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl"]
Oct 30 01:15:00.191731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.188651    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp"]
Oct 30 01:15:00.197746 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.194998    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl"
Oct 30 01:15:00.203205 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.203181    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp"
Oct 30 01:15:00.205959 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.203946    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw"]
Oct 30 01:15:00.205959 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.203996    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr"]
Oct 30 01:15:00.226745 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.223733    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv"]
Oct 30 01:15:00.230286 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.230262    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv"
Oct 30 01:15:00.230833 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.230812    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp"]
Oct 30 01:15:00.253069 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.253038    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6krwp\" (UniqueName: \"kubernetes.io/projected/75974319-3673-4bdc-bb2c-e9be475a5723-kube-api-access-6krwp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr\" (UID: \"75974319-3673-4bdc-bb2c-e9be475a5723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr"
Oct 30 01:15:00.253210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.253127    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c09d93-afed-4328-b7a4-23775614c03c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw\" (UID: \"07c09d93-afed-4328-b7a4-23775614c03c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw"
Oct 30 01:15:00.253372 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.253346    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75974319-3673-4bdc-bb2c-e9be475a5723-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr\" (UID: \"75974319-3673-4bdc-bb2c-e9be475a5723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr"
Oct 30 01:15:00.253468 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.253390    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwmh9\" (UniqueName: \"kubernetes.io/projected/07c09d93-afed-4328-b7a4-23775614c03c-kube-api-access-bwmh9\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw\" (UID: \"07c09d93-afed-4328-b7a4-23775614c03c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw"
Oct 30 01:15:00.265917 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.265890    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:15:00.279651 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.276414    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f0f160f-9861-48e1-8a8d-6e33f9a46a46" path="/var/lib/kubelet/pods/5f0f160f-9861-48e1-8a8d-6e33f9a46a46/volumes"
Oct 30 01:15:00.281573 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.281543    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl"]
Oct 30 01:15:00.298725 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.295187    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d"]
Oct 30 01:15:00.305938 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.301684    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79"]
Oct 30 01:15:00.305938 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.301856    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d"
Oct 30 01:15:00.308270 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.308247    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv"]
Oct 30 01:15:00.308392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.308369    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79"
Oct 30 01:15:00.309281 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.309261    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg"]
Oct 30 01:15:00.315912 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.315889    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d"]
Oct 30 01:15:00.316025 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.315982    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg"
Oct 30 01:15:00.321586 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.321565    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc"]
Oct 30 01:15:00.328046 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.326936    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79"]
Oct 30 01:15:00.328046 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.327009    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc"
Oct 30 01:15:00.345958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.331356    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg"]
Oct 30 01:15:00.345958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.342414    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc"]
Oct 30 01:15:00.361115 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.361081    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz7xz\" (UniqueName: \"kubernetes.io/projected/dc4a7cce-cbf8-43b2-99cd-e7a82ab54824-kube-api-access-sz7xz\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl\" (UID: \"dc4a7cce-cbf8-43b2-99cd-e7a82ab54824\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl"
Oct 30 01:15:00.361281 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.361175    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b64df5bf-b9b0-47f9-bbd5-7c2839b42a53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv\" (UID: \"b64df5bf-b9b0-47f9-bbd5-7c2839b42a53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv"
Oct 30 01:15:00.361281 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.361259    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c09d93-afed-4328-b7a4-23775614c03c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw\" (UID: \"07c09d93-afed-4328-b7a4-23775614c03c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw"
Oct 30 01:15:00.361430 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.361388    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn247\" (UniqueName: \"kubernetes.io/projected/b64df5bf-b9b0-47f9-bbd5-7c2839b42a53-kube-api-access-qn247\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv\" (UID: \"b64df5bf-b9b0-47f9-bbd5-7c2839b42a53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv"
Oct 30 01:15:00.361494 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.361453    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4a7cce-cbf8-43b2-99cd-e7a82ab54824-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl\" (UID: \"dc4a7cce-cbf8-43b2-99cd-e7a82ab54824\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl"
Oct 30 01:15:00.361560 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.361515    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffde68d-ddf9-439e-9cff-b0aab29c397e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp\" (UID: \"9ffde68d-ddf9-439e-9cff-b0aab29c397e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp"
Oct 30 01:15:00.361624 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.361612    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75974319-3673-4bdc-bb2c-e9be475a5723-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr\" (UID: \"75974319-3673-4bdc-bb2c-e9be475a5723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr"
Oct 30 01:15:00.361686 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.361652    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwmh9\" (UniqueName: \"kubernetes.io/projected/07c09d93-afed-4328-b7a4-23775614c03c-kube-api-access-bwmh9\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw\" (UID: \"07c09d93-afed-4328-b7a4-23775614c03c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw"
Oct 30 01:15:00.362789 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.362731    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75974319-3673-4bdc-bb2c-e9be475a5723-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr\" (UID: \"75974319-3673-4bdc-bb2c-e9be475a5723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr"
Oct 30 01:15:00.363794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.363768    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c09d93-afed-4328-b7a4-23775614c03c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw\" (UID: \"07c09d93-afed-4328-b7a4-23775614c03c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw"
Oct 30 01:15:00.363900 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.363853    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq5g2\" (UniqueName: \"kubernetes.io/projected/9ffde68d-ddf9-439e-9cff-b0aab29c397e-kube-api-access-vq5g2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp\" (UID: \"9ffde68d-ddf9-439e-9cff-b0aab29c397e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp"
Oct 30 01:15:00.364201 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.364166    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6krwp\" (UniqueName: \"kubernetes.io/projected/75974319-3673-4bdc-bb2c-e9be475a5723-kube-api-access-6krwp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr\" (UID: \"75974319-3673-4bdc-bb2c-e9be475a5723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr"
Oct 30 01:15:00.394902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.394880    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwmh9\" (UniqueName: \"kubernetes.io/projected/07c09d93-afed-4328-b7a4-23775614c03c-kube-api-access-bwmh9\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw\" (UID: \"07c09d93-afed-4328-b7a4-23775614c03c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw"
Oct 30 01:15:00.395044 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.394997    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6krwp\" (UniqueName: \"kubernetes.io/projected/75974319-3673-4bdc-bb2c-e9be475a5723-kube-api-access-6krwp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr\" (UID: \"75974319-3673-4bdc-bb2c-e9be475a5723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr"
Oct 30 01:15:00.465971 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.465893    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf83038-adae-454f-81e4-5ae0e19c660b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79\" (UID: \"2cf83038-adae-454f-81e4-5ae0e19c660b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79"
Oct 30 01:15:00.465971 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.465935    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h79ks\" (UniqueName: \"kubernetes.io/projected/5cca91cc-d895-4aa2-b636-8d8dd97cdbb8-kube-api-access-h79ks\") pod \"maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d\" (UID: \"5cca91cc-d895-4aa2-b636-8d8dd97cdbb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d"
Oct 30 01:15:00.466182 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.466043    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq5g2\" (UniqueName: \"kubernetes.io/projected/9ffde68d-ddf9-439e-9cff-b0aab29c397e-kube-api-access-vq5g2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp\" (UID: \"9ffde68d-ddf9-439e-9cff-b0aab29c397e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp"
Oct 30 01:15:00.466182 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.466067    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tb2f\" (UniqueName: \"kubernetes.io/projected/0b9a2818-c031-4bae-b1a2-d8793f88c00f-kube-api-access-4tb2f\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg\" (UID: \"0b9a2818-c031-4bae-b1a2-d8793f88c00f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg"
Oct 30 01:15:00.466266 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.466198    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c9e6dd5-8118-494e-8cfd-bab30c938ddc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc\" (UID: \"7c9e6dd5-8118-494e-8cfd-bab30c938ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc"
Oct 30 01:15:00.466266 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.466219    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc7qb\" (UniqueName: \"kubernetes.io/projected/7c9e6dd5-8118-494e-8cfd-bab30c938ddc-kube-api-access-fc7qb\") pod \"maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc\" (UID: \"7c9e6dd5-8118-494e-8cfd-bab30c938ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc"
Oct 30 01:15:00.466408 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.466385    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4hz2\" (UniqueName: \"kubernetes.io/projected/2cf83038-adae-454f-81e4-5ae0e19c660b-kube-api-access-t4hz2\") pod \"maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79\" (UID: \"2cf83038-adae-454f-81e4-5ae0e19c660b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79"
Oct 30 01:15:00.466528 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.466510    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9a2818-c031-4bae-b1a2-d8793f88c00f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg\" (UID: \"0b9a2818-c031-4bae-b1a2-d8793f88c00f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg"
Oct 30 01:15:00.466593 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.466563    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz7xz\" (UniqueName: \"kubernetes.io/projected/dc4a7cce-cbf8-43b2-99cd-e7a82ab54824-kube-api-access-sz7xz\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl\" (UID: \"dc4a7cce-cbf8-43b2-99cd-e7a82ab54824\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl"
Oct 30 01:15:00.466651 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.466606    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b64df5bf-b9b0-47f9-bbd5-7c2839b42a53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv\" (UID: \"b64df5bf-b9b0-47f9-bbd5-7c2839b42a53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv"
Oct 30 01:15:00.466752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.466735    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn247\" (UniqueName: \"kubernetes.io/projected/b64df5bf-b9b0-47f9-bbd5-7c2839b42a53-kube-api-access-qn247\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv\" (UID: \"b64df5bf-b9b0-47f9-bbd5-7c2839b42a53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv"
Oct 30 01:15:00.466844 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.466795    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4a7cce-cbf8-43b2-99cd-e7a82ab54824-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl\" (UID: \"dc4a7cce-cbf8-43b2-99cd-e7a82ab54824\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl"
Oct 30 01:15:00.466907 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.466827    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cca91cc-d895-4aa2-b636-8d8dd97cdbb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d\" (UID: \"5cca91cc-d895-4aa2-b636-8d8dd97cdbb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d"
Oct 30 01:15:00.466964 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.466900    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffde68d-ddf9-439e-9cff-b0aab29c397e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp\" (UID: \"9ffde68d-ddf9-439e-9cff-b0aab29c397e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp"
Oct 30 01:15:00.467323 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.467304    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b64df5bf-b9b0-47f9-bbd5-7c2839b42a53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv\" (UID: \"b64df5bf-b9b0-47f9-bbd5-7c2839b42a53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv"
Oct 30 01:15:00.467396 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.467379    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4a7cce-cbf8-43b2-99cd-e7a82ab54824-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl\" (UID: \"dc4a7cce-cbf8-43b2-99cd-e7a82ab54824\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl"
Oct 30 01:15:00.468518 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.468494    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffde68d-ddf9-439e-9cff-b0aab29c397e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp\" (UID: \"9ffde68d-ddf9-439e-9cff-b0aab29c397e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp"
Oct 30 01:15:00.476210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.476190    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw"
Oct 30 01:15:00.489075 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.489053    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz7xz\" (UniqueName: \"kubernetes.io/projected/dc4a7cce-cbf8-43b2-99cd-e7a82ab54824-kube-api-access-sz7xz\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl\" (UID: \"dc4a7cce-cbf8-43b2-99cd-e7a82ab54824\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl"
Oct 30 01:15:00.489165 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.489110    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq5g2\" (UniqueName: \"kubernetes.io/projected/9ffde68d-ddf9-439e-9cff-b0aab29c397e-kube-api-access-vq5g2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp\" (UID: \"9ffde68d-ddf9-439e-9cff-b0aab29c397e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp"
Oct 30 01:15:00.490487 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.490470    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn247\" (UniqueName: \"kubernetes.io/projected/b64df5bf-b9b0-47f9-bbd5-7c2839b42a53-kube-api-access-qn247\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv\" (UID: \"b64df5bf-b9b0-47f9-bbd5-7c2839b42a53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv"
Oct 30 01:15:00.534464 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.534418    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr"
Oct 30 01:15:00.536955 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.536930    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp"
Oct 30 01:15:00.547158 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.547133    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl"
Oct 30 01:15:00.569021 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.567726    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv"
Oct 30 01:15:00.570867 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.570216    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cca91cc-d895-4aa2-b636-8d8dd97cdbb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d\" (UID: \"5cca91cc-d895-4aa2-b636-8d8dd97cdbb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d"
Oct 30 01:15:00.570867 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.570369    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf83038-adae-454f-81e4-5ae0e19c660b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79\" (UID: \"2cf83038-adae-454f-81e4-5ae0e19c660b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79"
Oct 30 01:15:00.570867 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.570406    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h79ks\" (UniqueName: \"kubernetes.io/projected/5cca91cc-d895-4aa2-b636-8d8dd97cdbb8-kube-api-access-h79ks\") pod \"maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d\" (UID: \"5cca91cc-d895-4aa2-b636-8d8dd97cdbb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d"
Oct 30 01:15:00.570867 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.570460    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tb2f\" (UniqueName: \"kubernetes.io/projected/0b9a2818-c031-4bae-b1a2-d8793f88c00f-kube-api-access-4tb2f\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg\" (UID: \"0b9a2818-c031-4bae-b1a2-d8793f88c00f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg"
Oct 30 01:15:00.573728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.571230    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf83038-adae-454f-81e4-5ae0e19c660b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79\" (UID: \"2cf83038-adae-454f-81e4-5ae0e19c660b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79"
Oct 30 01:15:00.579158 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.578965    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c9e6dd5-8118-494e-8cfd-bab30c938ddc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc\" (UID: \"7c9e6dd5-8118-494e-8cfd-bab30c938ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc"
Oct 30 01:15:00.579158 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.579007    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc7qb\" (UniqueName: \"kubernetes.io/projected/7c9e6dd5-8118-494e-8cfd-bab30c938ddc-kube-api-access-fc7qb\") pod \"maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc\" (UID: \"7c9e6dd5-8118-494e-8cfd-bab30c938ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc"
Oct 30 01:15:00.579158 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.579100    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4hz2\" (UniqueName: \"kubernetes.io/projected/2cf83038-adae-454f-81e4-5ae0e19c660b-kube-api-access-t4hz2\") pod \"maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79\" (UID: \"2cf83038-adae-454f-81e4-5ae0e19c660b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79"
Oct 30 01:15:00.579384 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.579209    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9a2818-c031-4bae-b1a2-d8793f88c00f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg\" (UID: \"0b9a2818-c031-4bae-b1a2-d8793f88c00f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg"
Oct 30 01:15:00.581691 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.580519    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cca91cc-d895-4aa2-b636-8d8dd97cdbb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d\" (UID: \"5cca91cc-d895-4aa2-b636-8d8dd97cdbb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d"
Oct 30 01:15:00.581691 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.581673    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c9e6dd5-8118-494e-8cfd-bab30c938ddc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc\" (UID: \"7c9e6dd5-8118-494e-8cfd-bab30c938ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc"
Oct 30 01:15:00.583990 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.582574    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9a2818-c031-4bae-b1a2-d8793f88c00f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg\" (UID: \"0b9a2818-c031-4bae-b1a2-d8793f88c00f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg"
Oct 30 01:15:00.606308 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.606281    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc7qb\" (UniqueName: \"kubernetes.io/projected/7c9e6dd5-8118-494e-8cfd-bab30c938ddc-kube-api-access-fc7qb\") pod \"maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc\" (UID: \"7c9e6dd5-8118-494e-8cfd-bab30c938ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc"
Oct 30 01:15:00.606414 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.606286    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4hz2\" (UniqueName: \"kubernetes.io/projected/2cf83038-adae-454f-81e4-5ae0e19c660b-kube-api-access-t4hz2\") pod \"maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79\" (UID: \"2cf83038-adae-454f-81e4-5ae0e19c660b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79"
Oct 30 01:15:00.608963 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.608925    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tb2f\" (UniqueName: \"kubernetes.io/projected/0b9a2818-c031-4bae-b1a2-d8793f88c00f-kube-api-access-4tb2f\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg\" (UID: \"0b9a2818-c031-4bae-b1a2-d8793f88c00f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg"
Oct 30 01:15:00.611371 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.611346    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h79ks\" (UniqueName: \"kubernetes.io/projected/5cca91cc-d895-4aa2-b636-8d8dd97cdbb8-kube-api-access-h79ks\") pod \"maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d\" (UID: \"5cca91cc-d895-4aa2-b636-8d8dd97cdbb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d"
Oct 30 01:15:00.658050 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.658008    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d"
Oct 30 01:15:00.694054 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.693820    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg"
Oct 30 01:15:00.824728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.820248    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79"
Oct 30 01:15:00.824728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.822910    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc"
Oct 30 01:15:00.956361 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:00.956326    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw"]
Oct 30 01:15:01.350750 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:01.350653    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr"]
Oct 30 01:15:01.351295 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:15:01.350780    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75974319_3673_4bdc_bb2c_e9be475a5723.slice/crio-3b6030897464383cf55629bcbcc58ff613282241d4dad12e2fd04fce562b305b WatchSource:0}: Error finding container 3b6030897464383cf55629bcbcc58ff613282241d4dad12e2fd04fce562b305b: Status 404 returned error can't find the container with id 3b6030897464383cf55629bcbcc58ff613282241d4dad12e2fd04fce562b305b
Oct 30 01:15:01.427308 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:01.403979    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv"]
Oct 30 01:15:01.427308 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:15:01.405570    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64df5bf_b9b0_47f9_bbd5_7c2839b42a53.slice/crio-af0b0a5685ec99b0f6ddb2452ab70bb0955ad7a96d9522d35c850245e45d20ef WatchSource:0}: Error finding container af0b0a5685ec99b0f6ddb2452ab70bb0955ad7a96d9522d35c850245e45d20ef: Status 404 returned error can't find the container with id af0b0a5685ec99b0f6ddb2452ab70bb0955ad7a96d9522d35c850245e45d20ef
Oct 30 01:15:01.427308 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:01.407630    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw" event={"ID":"07c09d93-afed-4328-b7a4-23775614c03c","Type":"ContainerStarted","Data":"3c832147d0de36ac5098dc30e9fbac585fb2cdb26bcc690e5282541e302e7903"}
Oct 30 01:15:01.427308 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:01.409880    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr" event={"ID":"75974319-3673-4bdc-bb2c-e9be475a5723","Type":"ContainerStarted","Data":"3b6030897464383cf55629bcbcc58ff613282241d4dad12e2fd04fce562b305b"}
Oct 30 01:15:01.508393 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:01.507991    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp"]
Oct 30 01:15:01.525228 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:01.522333    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg"]
Oct 30 01:15:01.529003 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:01.528975    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl"]
Oct 30 01:15:01.532285 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:15:01.530096    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffde68d_ddf9_439e_9cff_b0aab29c397e.slice/crio-7ca1948ff2cee6aa501935f869f5e08d3f926bfa03b76e3f171ad67af98c4291 WatchSource:0}: Error finding container 7ca1948ff2cee6aa501935f869f5e08d3f926bfa03b76e3f171ad67af98c4291: Status 404 returned error can't find the container with id 7ca1948ff2cee6aa501935f869f5e08d3f926bfa03b76e3f171ad67af98c4291
Oct 30 01:15:01.548064 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:15:01.548040    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4a7cce_cbf8_43b2_99cd_e7a82ab54824.slice/crio-cc93a47aa9f7ef791b05b8763304b338323b7317c301189707f0388571d0ce2d WatchSource:0}: Error finding container cc93a47aa9f7ef791b05b8763304b338323b7317c301189707f0388571d0ce2d: Status 404 returned error can't find the container with id cc93a47aa9f7ef791b05b8763304b338323b7317c301189707f0388571d0ce2d
Oct 30 01:15:01.644251 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:15:01.644220    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cca91cc_d895_4aa2_b636_8d8dd97cdbb8.slice/crio-15003d4d1fc11cd9e24bc9fcd52d6143fb342b944c1248aab4e5802b9c94ff99 WatchSource:0}: Error finding container 15003d4d1fc11cd9e24bc9fcd52d6143fb342b944c1248aab4e5802b9c94ff99: Status 404 returned error can't find the container with id 15003d4d1fc11cd9e24bc9fcd52d6143fb342b944c1248aab4e5802b9c94ff99
Oct 30 01:15:01.645193 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:01.645162    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d"]
Oct 30 01:15:01.655481 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:01.652094    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc"]
Oct 30 01:15:01.999163 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:01.993749    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79"]
Oct 30 01:15:01.999163 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:15:01.996295    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf83038_adae_454f_81e4_5ae0e19c660b.slice/crio-8838bf645cb96c50bc8ada0bbc8c3c6bda5c393739ea37cb74cbdc20fad1765a WatchSource:0}: Error finding container 8838bf645cb96c50bc8ada0bbc8c3c6bda5c393739ea37cb74cbdc20fad1765a: Status 404 returned error can't find the container with id 8838bf645cb96c50bc8ada0bbc8c3c6bda5c393739ea37cb74cbdc20fad1765a
Oct 30 01:15:02.515935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:02.515153    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw" event={"ID":"07c09d93-afed-4328-b7a4-23775614c03c","Type":"ContainerStarted","Data":"2e1f06c0d088e9ff80bb14e97243b95dc5d9d78b78da229111af16d9118d9167"}
Oct 30 01:15:02.522233 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:02.522203    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr" event={"ID":"75974319-3673-4bdc-bb2c-e9be475a5723","Type":"ContainerStarted","Data":"55484a63c15106296ee66f03a105ee15a89cf6e09007c76261362f63ae29abd6"}
Oct 30 01:15:02.525076 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:02.525053    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg" event={"ID":"0b9a2818-c031-4bae-b1a2-d8793f88c00f","Type":"ContainerStarted","Data":"be8a62facdfc66387cdc864f567c09530124d356ac17d40ec4db5b2576b04732"}
Oct 30 01:15:02.525244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:02.525228    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg" event={"ID":"0b9a2818-c031-4bae-b1a2-d8793f88c00f","Type":"ContainerStarted","Data":"2ae525b62c4280eee763c591887b0b2784003ed82a05454e0741f795a9402ea7"}
Oct 30 01:15:02.530094 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:02.530036    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp" event={"ID":"9ffde68d-ddf9-439e-9cff-b0aab29c397e","Type":"ContainerStarted","Data":"7ca1948ff2cee6aa501935f869f5e08d3f926bfa03b76e3f171ad67af98c4291"}
Oct 30 01:15:02.531685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:02.531665    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl" event={"ID":"dc4a7cce-cbf8-43b2-99cd-e7a82ab54824","Type":"ContainerStarted","Data":"43909fac568e608223dba80804ca8f02a813041c9f254a8acd541f7af5d7ed92"}
Oct 30 01:15:02.531813 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:02.531721    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl" event={"ID":"dc4a7cce-cbf8-43b2-99cd-e7a82ab54824","Type":"ContainerStarted","Data":"cc93a47aa9f7ef791b05b8763304b338323b7317c301189707f0388571d0ce2d"}
Oct 30 01:15:02.533855 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:02.533805    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d" event={"ID":"5cca91cc-d895-4aa2-b636-8d8dd97cdbb8","Type":"ContainerStarted","Data":"15003d4d1fc11cd9e24bc9fcd52d6143fb342b944c1248aab4e5802b9c94ff99"}
Oct 30 01:15:02.535575 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:02.535527    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv" event={"ID":"b64df5bf-b9b0-47f9-bbd5-7c2839b42a53","Type":"ContainerStarted","Data":"16c2212c484f114d26febd73827ea3e1dcad4a0bf69579a2257410246e7afeb3"}
Oct 30 01:15:02.535668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:02.535592    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv" event={"ID":"b64df5bf-b9b0-47f9-bbd5-7c2839b42a53","Type":"ContainerStarted","Data":"af0b0a5685ec99b0f6ddb2452ab70bb0955ad7a96d9522d35c850245e45d20ef"}
Oct 30 01:15:02.537329 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:02.537302    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc" event={"ID":"7c9e6dd5-8118-494e-8cfd-bab30c938ddc","Type":"ContainerStarted","Data":"e12bc6ab53e2ff994c26d546d688415df4518d53282d1edacfea441d16afb533"}
Oct 30 01:15:02.538527 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:02.538510    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79" event={"ID":"2cf83038-adae-454f-81e4-5ae0e19c660b","Type":"ContainerStarted","Data":"8838bf645cb96c50bc8ada0bbc8c3c6bda5c393739ea37cb74cbdc20fad1765a"}
Oct 30 01:15:02.563744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:02.563673    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw" podStartSLOduration=2.563659781 podStartE2EDuration="2.563659781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.558361861 +0000 UTC m=+2396.415269608" watchObservedRunningTime="2026-10-30 01:15:02.563659781 +0000 UTC m=+2396.420567538"
Oct 30 01:15:02.604398 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:02.604351    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl" podStartSLOduration=2.604336441 podStartE2EDuration="2.604336441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.599290889 +0000 UTC m=+2396.456198648" watchObservedRunningTime="2026-10-30 01:15:02.604336441 +0000 UTC m=+2396.461244251"
Oct 30 01:15:02.638431 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:02.638376    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv" podStartSLOduration=2.638361422 podStartE2EDuration="2.638361422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.637620989 +0000 UTC m=+2396.494528746" watchObservedRunningTime="2026-10-30 01:15:02.638361422 +0000 UTC m=+2396.495269402"
Oct 30 01:15:02.679659 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:02.679596    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr" podStartSLOduration=2.679578122 podStartE2EDuration="2.679578122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.679521455 +0000 UTC m=+2396.536429211" watchObservedRunningTime="2026-10-30 01:15:02.679578122 +0000 UTC m=+2396.536485878"
Oct 30 01:15:02.719975 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:02.719927    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg" podStartSLOduration=2.719915056 podStartE2EDuration="2.719915056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.717526565 +0000 UTC m=+2396.574434320" watchObservedRunningTime="2026-10-30 01:15:02.719915056 +0000 UTC m=+2396.576822807"
Oct 30 01:15:03.549300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.549266    6099 generic.go:356] "Generic (PLEG): container finished" podID="b64df5bf-b9b0-47f9-bbd5-7c2839b42a53" containerID="16c2212c484f114d26febd73827ea3e1dcad4a0bf69579a2257410246e7afeb3" exitCode=60
Oct 30 01:15:03.549692 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.549343    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv" event={"ID":"b64df5bf-b9b0-47f9-bbd5-7c2839b42a53","Type":"ContainerDied","Data":"16c2212c484f114d26febd73827ea3e1dcad4a0bf69579a2257410246e7afeb3"}
Oct 30 01:15:03.549692 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.549676    6099 scope.go:122] "RemoveContainer" containerID="16c2212c484f114d26febd73827ea3e1dcad4a0bf69579a2257410246e7afeb3"
Oct 30 01:15:03.551137 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.551115    6099 generic.go:356] "Generic (PLEG): container finished" podID="7c9e6dd5-8118-494e-8cfd-bab30c938ddc" containerID="cf62bea2a0f3ad26ea2cb51391ec10d38e9e928b6f1ffb91f4df5db357895a39" exitCode=60
Oct 30 01:15:03.551267 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.551143    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc" event={"ID":"7c9e6dd5-8118-494e-8cfd-bab30c938ddc","Type":"ContainerDied","Data":"cf62bea2a0f3ad26ea2cb51391ec10d38e9e928b6f1ffb91f4df5db357895a39"}
Oct 30 01:15:03.551665 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.551467    6099 scope.go:122] "RemoveContainer" containerID="cf62bea2a0f3ad26ea2cb51391ec10d38e9e928b6f1ffb91f4df5db357895a39"
Oct 30 01:15:03.559383 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.559364    6099 generic.go:356] "Generic (PLEG): container finished" podID="2cf83038-adae-454f-81e4-5ae0e19c660b" containerID="f69f1f279e00090f14316274106df35e4a304cdddcbca41ea609ed63b4513989" exitCode=60
Oct 30 01:15:03.559475 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.559452    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79" event={"ID":"2cf83038-adae-454f-81e4-5ae0e19c660b","Type":"ContainerDied","Data":"f69f1f279e00090f14316274106df35e4a304cdddcbca41ea609ed63b4513989"}
Oct 30 01:15:03.559894 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.559752    6099 scope.go:122] "RemoveContainer" containerID="f69f1f279e00090f14316274106df35e4a304cdddcbca41ea609ed63b4513989"
Oct 30 01:15:03.561544 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.561524    6099 generic.go:356] "Generic (PLEG): container finished" podID="07c09d93-afed-4328-b7a4-23775614c03c" containerID="2e1f06c0d088e9ff80bb14e97243b95dc5d9d78b78da229111af16d9118d9167" exitCode=60
Oct 30 01:15:03.561628 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.561566    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw" event={"ID":"07c09d93-afed-4328-b7a4-23775614c03c","Type":"ContainerDied","Data":"2e1f06c0d088e9ff80bb14e97243b95dc5d9d78b78da229111af16d9118d9167"}
Oct 30 01:15:03.561843 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.561827    6099 scope.go:122] "RemoveContainer" containerID="2e1f06c0d088e9ff80bb14e97243b95dc5d9d78b78da229111af16d9118d9167"
Oct 30 01:15:03.563342 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.563320    6099 generic.go:356] "Generic (PLEG): container finished" podID="75974319-3673-4bdc-bb2c-e9be475a5723" containerID="55484a63c15106296ee66f03a105ee15a89cf6e09007c76261362f63ae29abd6" exitCode=60
Oct 30 01:15:03.563439 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.563393    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr" event={"ID":"75974319-3673-4bdc-bb2c-e9be475a5723","Type":"ContainerDied","Data":"55484a63c15106296ee66f03a105ee15a89cf6e09007c76261362f63ae29abd6"}
Oct 30 01:15:03.563610 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.563594    6099 scope.go:122] "RemoveContainer" containerID="55484a63c15106296ee66f03a105ee15a89cf6e09007c76261362f63ae29abd6"
Oct 30 01:15:03.565317 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.565293    6099 generic.go:356] "Generic (PLEG): container finished" podID="0b9a2818-c031-4bae-b1a2-d8793f88c00f" containerID="be8a62facdfc66387cdc864f567c09530124d356ac17d40ec4db5b2576b04732" exitCode=60
Oct 30 01:15:03.565422 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.565367    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg" event={"ID":"0b9a2818-c031-4bae-b1a2-d8793f88c00f","Type":"ContainerDied","Data":"be8a62facdfc66387cdc864f567c09530124d356ac17d40ec4db5b2576b04732"}
Oct 30 01:15:03.565607 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.565592    6099 scope.go:122] "RemoveContainer" containerID="be8a62facdfc66387cdc864f567c09530124d356ac17d40ec4db5b2576b04732"
Oct 30 01:15:03.567104 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.567084    6099 generic.go:356] "Generic (PLEG): container finished" podID="9ffde68d-ddf9-439e-9cff-b0aab29c397e" containerID="295c554c2e4a86180e08b89d9569dc3fbe79208295d4ef834c7ee6d9d6d5ca37" exitCode=60
Oct 30 01:15:03.567206 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.567109    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp" event={"ID":"9ffde68d-ddf9-439e-9cff-b0aab29c397e","Type":"ContainerDied","Data":"295c554c2e4a86180e08b89d9569dc3fbe79208295d4ef834c7ee6d9d6d5ca37"}
Oct 30 01:15:03.567365 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.567350    6099 scope.go:122] "RemoveContainer" containerID="295c554c2e4a86180e08b89d9569dc3fbe79208295d4ef834c7ee6d9d6d5ca37"
Oct 30 01:15:03.569445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.569423    6099 generic.go:356] "Generic (PLEG): container finished" podID="dc4a7cce-cbf8-43b2-99cd-e7a82ab54824" containerID="43909fac568e608223dba80804ca8f02a813041c9f254a8acd541f7af5d7ed92" exitCode=60
Oct 30 01:15:03.569870 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.569646    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl" event={"ID":"dc4a7cce-cbf8-43b2-99cd-e7a82ab54824","Type":"ContainerDied","Data":"43909fac568e608223dba80804ca8f02a813041c9f254a8acd541f7af5d7ed92"}
Oct 30 01:15:03.570414 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.570194    6099 scope.go:122] "RemoveContainer" containerID="43909fac568e608223dba80804ca8f02a813041c9f254a8acd541f7af5d7ed92"
Oct 30 01:15:03.572375 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.572354    6099 generic.go:356] "Generic (PLEG): container finished" podID="5cca91cc-d895-4aa2-b636-8d8dd97cdbb8" containerID="1a55e07a81ce1bb0d93ad786727a3605300f0b40e32df1712b021df50d8d4457" exitCode=60
Oct 30 01:15:03.572484 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.572390    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d" event={"ID":"5cca91cc-d895-4aa2-b636-8d8dd97cdbb8","Type":"ContainerDied","Data":"1a55e07a81ce1bb0d93ad786727a3605300f0b40e32df1712b021df50d8d4457"}
Oct 30 01:15:03.572679 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:03.572662    6099 scope.go:122] "RemoveContainer" containerID="1a55e07a81ce1bb0d93ad786727a3605300f0b40e32df1712b021df50d8d4457"
Oct 30 01:15:04.583758 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:04.583683    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw" event={"ID":"07c09d93-afed-4328-b7a4-23775614c03c","Type":"ContainerStarted","Data":"0a7d134ffa34b710a5af79a4a03121437ee052f41c70ff3dbfb0788f78b701a3"}
Oct 30 01:15:04.585848 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:04.585826    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr" event={"ID":"75974319-3673-4bdc-bb2c-e9be475a5723","Type":"ContainerStarted","Data":"3edd280585a4c94c4ba1dcbdcf8da2c1ccb86e7de7cc2248e82a90c5bbb15947"}
Oct 30 01:15:04.587880 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:04.587855    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg" event={"ID":"0b9a2818-c031-4bae-b1a2-d8793f88c00f","Type":"ContainerStarted","Data":"e61d4b7eace3841a34adc5f2528c7fe23e290cf7e8a6bba6c8c46ebd80d2e4ca"}
Oct 30 01:15:04.589814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:04.589793    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp" event={"ID":"9ffde68d-ddf9-439e-9cff-b0aab29c397e","Type":"ContainerStarted","Data":"810383acd317e0cdd2c663c65a4fa34652dcd39cb5c6e00640c5e8246e7f841f"}
Oct 30 01:15:04.591983 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:04.591964    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl" event={"ID":"dc4a7cce-cbf8-43b2-99cd-e7a82ab54824","Type":"ContainerStarted","Data":"ed6ecaff58545012c2585eb9a7badefd8b30fd6f1804fc17f9793beb1762ed89"}
Oct 30 01:15:04.593917 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:04.593895    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d" event={"ID":"5cca91cc-d895-4aa2-b636-8d8dd97cdbb8","Type":"ContainerStarted","Data":"f9dc67452dfda7a4c33e0c02516cd880c08dda9ccf818b94a207669aec4c9e30"}
Oct 30 01:15:04.595719 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:04.595688    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv" event={"ID":"b64df5bf-b9b0-47f9-bbd5-7c2839b42a53","Type":"ContainerStarted","Data":"0032769f338d729fd9ea09b5f755f1532c1f6361404006ab4c1d24fa1761f931"}
Oct 30 01:15:04.597674 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:04.597656    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc" event={"ID":"7c9e6dd5-8118-494e-8cfd-bab30c938ddc","Type":"ContainerStarted","Data":"dda707756df08615c284922e27a30c699641e8c1b141975d9b4a3572758212b0"}
Oct 30 01:15:04.610251 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:04.610218    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79" event={"ID":"2cf83038-adae-454f-81e4-5ae0e19c660b","Type":"ContainerStarted","Data":"01310b226242c0b3963db84f19c2ae6b25a73eaf85f688a5035c25d0bc1ea158"}
Oct 30 01:15:04.697154 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:04.697098    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp" podStartSLOduration=4.697083918 podStartE2EDuration="4.697083918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.691898563 +0000 UTC m=+2398.548806319" watchObservedRunningTime="2026-10-30 01:15:04.697083918 +0000 UTC m=+2398.553991681"
Oct 30 01:15:04.781617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:04.781557    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79" podStartSLOduration=4.781538533 podStartE2EDuration="4.781538533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.777447201 +0000 UTC m=+2398.634354957" watchObservedRunningTime="2026-10-30 01:15:04.781538533 +0000 UTC m=+2398.638446289"
Oct 30 01:15:04.862988 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:04.862938    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc" podStartSLOduration=4.862925303 podStartE2EDuration="4.862925303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.859625287 +0000 UTC m=+2398.716533043" watchObservedRunningTime="2026-10-30 01:15:04.862925303 +0000 UTC m=+2398.719833057"
Oct 30 01:15:04.905384 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:04.905338    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d" podStartSLOduration=4.905323617 podStartE2EDuration="4.905323617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.90407502 +0000 UTC m=+2398.760982771" watchObservedRunningTime="2026-10-30 01:15:04.905323617 +0000 UTC m=+2398.762231417"
Oct 30 01:15:05.398647 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.398588    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2" podUID="9e96d7ed-1b43-45d9-9d84-64b9aaf7e694" containerName="main" containerID="cri-o://6ca883d51f797409e57035d6721e180878a99b5bd1b90e106acf32ca954cb302" gracePeriod=45
Oct 30 01:15:05.625077 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.625047    6099 generic.go:356] "Generic (PLEG): container finished" podID="07c09d93-afed-4328-b7a4-23775614c03c" containerID="0a7d134ffa34b710a5af79a4a03121437ee052f41c70ff3dbfb0788f78b701a3" exitCode=60
Oct 30 01:15:05.625543 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.625138    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw" event={"ID":"07c09d93-afed-4328-b7a4-23775614c03c","Type":"ContainerDied","Data":"0a7d134ffa34b710a5af79a4a03121437ee052f41c70ff3dbfb0788f78b701a3"}
Oct 30 01:15:05.625690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.625564    6099 scope.go:122] "RemoveContainer" containerID="0a7d134ffa34b710a5af79a4a03121437ee052f41c70ff3dbfb0788f78b701a3"
Oct 30 01:15:05.626070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.626051    6099 scope.go:122] "RemoveContainer" containerID="2e1f06c0d088e9ff80bb14e97243b95dc5d9d78b78da229111af16d9118d9167"
Oct 30 01:15:05.626146 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:05.626069    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw_odh-ai-gateway-infra(07c09d93-afed-4328-b7a4-23775614c03c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw" podUID="07c09d93-afed-4328-b7a4-23775614c03c"
Oct 30 01:15:05.628513 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.628488    6099 generic.go:356] "Generic (PLEG): container finished" podID="75974319-3673-4bdc-bb2c-e9be475a5723" containerID="3edd280585a4c94c4ba1dcbdcf8da2c1ccb86e7de7cc2248e82a90c5bbb15947" exitCode=60
Oct 30 01:15:05.628682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.628659    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr" event={"ID":"75974319-3673-4bdc-bb2c-e9be475a5723","Type":"ContainerDied","Data":"3edd280585a4c94c4ba1dcbdcf8da2c1ccb86e7de7cc2248e82a90c5bbb15947"}
Oct 30 01:15:05.628958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.628937    6099 scope.go:122] "RemoveContainer" containerID="3edd280585a4c94c4ba1dcbdcf8da2c1ccb86e7de7cc2248e82a90c5bbb15947"
Oct 30 01:15:05.629325 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:05.629307    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr_odh-ai-gateway-infra(75974319-3673-4bdc-bb2c-e9be475a5723)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr" podUID="75974319-3673-4bdc-bb2c-e9be475a5723"
Oct 30 01:15:05.631209 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.631185    6099 generic.go:356] "Generic (PLEG): container finished" podID="9e96d7ed-1b43-45d9-9d84-64b9aaf7e694" containerID="6ca883d51f797409e57035d6721e180878a99b5bd1b90e106acf32ca954cb302" exitCode=0
Oct 30 01:15:05.631305 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.631285    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2" event={"ID":"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694","Type":"ContainerDied","Data":"6ca883d51f797409e57035d6721e180878a99b5bd1b90e106acf32ca954cb302"}
Oct 30 01:15:05.634932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.634909    6099 generic.go:356] "Generic (PLEG): container finished" podID="0b9a2818-c031-4bae-b1a2-d8793f88c00f" containerID="e61d4b7eace3841a34adc5f2528c7fe23e290cf7e8a6bba6c8c46ebd80d2e4ca" exitCode=60
Oct 30 01:15:05.635125 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.634935    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg" event={"ID":"0b9a2818-c031-4bae-b1a2-d8793f88c00f","Type":"ContainerDied","Data":"e61d4b7eace3841a34adc5f2528c7fe23e290cf7e8a6bba6c8c46ebd80d2e4ca"}
Oct 30 01:15:05.635559 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.635281    6099 scope.go:122] "RemoveContainer" containerID="e61d4b7eace3841a34adc5f2528c7fe23e290cf7e8a6bba6c8c46ebd80d2e4ca"
Oct 30 01:15:05.635691 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:05.635644    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg_odh-ai-gateway-infra(0b9a2818-c031-4bae-b1a2-d8793f88c00f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg" podUID="0b9a2818-c031-4bae-b1a2-d8793f88c00f"
Oct 30 01:15:05.639000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.637867    6099 generic.go:356] "Generic (PLEG): container finished" podID="9ffde68d-ddf9-439e-9cff-b0aab29c397e" containerID="810383acd317e0cdd2c663c65a4fa34652dcd39cb5c6e00640c5e8246e7f841f" exitCode=60
Oct 30 01:15:05.639000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.638585    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp" event={"ID":"9ffde68d-ddf9-439e-9cff-b0aab29c397e","Type":"ContainerDied","Data":"810383acd317e0cdd2c663c65a4fa34652dcd39cb5c6e00640c5e8246e7f841f"}
Oct 30 01:15:05.639000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.638605    6099 scope.go:122] "RemoveContainer" containerID="810383acd317e0cdd2c663c65a4fa34652dcd39cb5c6e00640c5e8246e7f841f"
Oct 30 01:15:05.639000 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:05.638944    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp_odh-ai-gateway-infra(9ffde68d-ddf9-439e-9cff-b0aab29c397e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp" podUID="9ffde68d-ddf9-439e-9cff-b0aab29c397e"
Oct 30 01:15:05.641696 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.641675    6099 generic.go:356] "Generic (PLEG): container finished" podID="dc4a7cce-cbf8-43b2-99cd-e7a82ab54824" containerID="ed6ecaff58545012c2585eb9a7badefd8b30fd6f1804fc17f9793beb1762ed89" exitCode=60
Oct 30 01:15:05.641807 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.641789    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl" event={"ID":"dc4a7cce-cbf8-43b2-99cd-e7a82ab54824","Type":"ContainerDied","Data":"ed6ecaff58545012c2585eb9a7badefd8b30fd6f1804fc17f9793beb1762ed89"}
Oct 30 01:15:05.642052 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.642037    6099 scope.go:122] "RemoveContainer" containerID="ed6ecaff58545012c2585eb9a7badefd8b30fd6f1804fc17f9793beb1762ed89"
Oct 30 01:15:05.642269 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:05.642251    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl_odh-ai-gateway-infra(dc4a7cce-cbf8-43b2-99cd-e7a82ab54824)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl" podUID="dc4a7cce-cbf8-43b2-99cd-e7a82ab54824"
Oct 30 01:15:05.647007 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.646981    6099 generic.go:356] "Generic (PLEG): container finished" podID="5cca91cc-d895-4aa2-b636-8d8dd97cdbb8" containerID="f9dc67452dfda7a4c33e0c02516cd880c08dda9ccf818b94a207669aec4c9e30" exitCode=60
Oct 30 01:15:05.647167 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.647150    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d" event={"ID":"5cca91cc-d895-4aa2-b636-8d8dd97cdbb8","Type":"ContainerDied","Data":"f9dc67452dfda7a4c33e0c02516cd880c08dda9ccf818b94a207669aec4c9e30"}
Oct 30 01:15:05.647514 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.647494    6099 scope.go:122] "RemoveContainer" containerID="f9dc67452dfda7a4c33e0c02516cd880c08dda9ccf818b94a207669aec4c9e30"
Oct 30 01:15:05.647938 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:05.647908    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d_odh-ai-gateway-infra(5cca91cc-d895-4aa2-b636-8d8dd97cdbb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d" podUID="5cca91cc-d895-4aa2-b636-8d8dd97cdbb8"
Oct 30 01:15:05.649837 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.649808    6099 scope.go:122] "RemoveContainer" containerID="55484a63c15106296ee66f03a105ee15a89cf6e09007c76261362f63ae29abd6"
Oct 30 01:15:05.651304 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.651282    6099 generic.go:356] "Generic (PLEG): container finished" podID="b64df5bf-b9b0-47f9-bbd5-7c2839b42a53" containerID="0032769f338d729fd9ea09b5f755f1532c1f6361404006ab4c1d24fa1761f931" exitCode=60
Oct 30 01:15:05.651462 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.651402    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv" event={"ID":"b64df5bf-b9b0-47f9-bbd5-7c2839b42a53","Type":"ContainerDied","Data":"0032769f338d729fd9ea09b5f755f1532c1f6361404006ab4c1d24fa1761f931"}
Oct 30 01:15:05.651739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.651687    6099 scope.go:122] "RemoveContainer" containerID="0032769f338d729fd9ea09b5f755f1532c1f6361404006ab4c1d24fa1761f931"
Oct 30 01:15:05.652019 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:05.651991    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv_odh-ai-gateway-infra(b64df5bf-b9b0-47f9-bbd5-7c2839b42a53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv" podUID="b64df5bf-b9b0-47f9-bbd5-7c2839b42a53"
Oct 30 01:15:05.654901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.654871    6099 generic.go:356] "Generic (PLEG): container finished" podID="7c9e6dd5-8118-494e-8cfd-bab30c938ddc" containerID="dda707756df08615c284922e27a30c699641e8c1b141975d9b4a3572758212b0" exitCode=60
Oct 30 01:15:05.654901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.654896    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc" event={"ID":"7c9e6dd5-8118-494e-8cfd-bab30c938ddc","Type":"ContainerDied","Data":"dda707756df08615c284922e27a30c699641e8c1b141975d9b4a3572758212b0"}
Oct 30 01:15:05.655362 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.655182    6099 scope.go:122] "RemoveContainer" containerID="dda707756df08615c284922e27a30c699641e8c1b141975d9b4a3572758212b0"
Oct 30 01:15:05.655465 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:05.655438    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc_odh-ai-gateway-infra(7c9e6dd5-8118-494e-8cfd-bab30c938ddc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc" podUID="7c9e6dd5-8118-494e-8cfd-bab30c938ddc"
Oct 30 01:15:05.656994 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.656972    6099 generic.go:356] "Generic (PLEG): container finished" podID="2cf83038-adae-454f-81e4-5ae0e19c660b" containerID="01310b226242c0b3963db84f19c2ae6b25a73eaf85f688a5035c25d0bc1ea158" exitCode=60
Oct 30 01:15:05.657916 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.657021    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79" event={"ID":"2cf83038-adae-454f-81e4-5ae0e19c660b","Type":"ContainerDied","Data":"01310b226242c0b3963db84f19c2ae6b25a73eaf85f688a5035c25d0bc1ea158"}
Oct 30 01:15:05.657916 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.657288    6099 scope.go:122] "RemoveContainer" containerID="01310b226242c0b3963db84f19c2ae6b25a73eaf85f688a5035c25d0bc1ea158"
Oct 30 01:15:05.657916 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:05.657555    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79_odh-ai-gateway-infra(2cf83038-adae-454f-81e4-5ae0e19c660b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79" podUID="2cf83038-adae-454f-81e4-5ae0e19c660b"
Oct 30 01:15:05.700932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.700905    6099 scope.go:122] "RemoveContainer" containerID="be8a62facdfc66387cdc864f567c09530124d356ac17d40ec4db5b2576b04732"
Oct 30 01:15:05.703158 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.703136    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:15:05.719429 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.719408    6099 scope.go:122] "RemoveContainer" containerID="295c554c2e4a86180e08b89d9569dc3fbe79208295d4ef834c7ee6d9d6d5ca37"
Oct 30 01:15:05.739328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.739309    6099 scope.go:122] "RemoveContainer" containerID="43909fac568e608223dba80804ca8f02a813041c9f254a8acd541f7af5d7ed92"
Oct 30 01:15:05.753359 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.753338    6099 scope.go:122] "RemoveContainer" containerID="1a55e07a81ce1bb0d93ad786727a3605300f0b40e32df1712b021df50d8d4457"
Oct 30 01:15:05.767914 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.767884    6099 scope.go:122] "RemoveContainer" containerID="16c2212c484f114d26febd73827ea3e1dcad4a0bf69579a2257410246e7afeb3"
Oct 30 01:15:05.781936 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.781913    6099 scope.go:122] "RemoveContainer" containerID="cf62bea2a0f3ad26ea2cb51391ec10d38e9e928b6f1ffb91f4df5db357895a39"
Oct 30 01:15:05.796998 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.796976    6099 scope.go:122] "RemoveContainer" containerID="f69f1f279e00090f14316274106df35e4a304cdddcbca41ea609ed63b4513989"
Oct 30 01:15:05.836731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.836666    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-dshm\") pod \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") "
Oct 30 01:15:05.836838 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.836802    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-home\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-home\") pod \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") "
Oct 30 01:15:05.836944 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.836925    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-kube-api-access-vrp5b\" (UniqueName: \"kubernetes.io/projected/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-kube-api-access-vrp5b\") pod \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") "
Oct 30 01:15:05.837095 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.837071    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-model-cache\") pod \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") "
Oct 30 01:15:05.837217 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.837094    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-home" pod "9e96d7ed-1b43-45d9-9d84-64b9aaf7e694" (UID: "9e96d7ed-1b43-45d9-9d84-64b9aaf7e694"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:05.837325 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.837304    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-model-cache" pod "9e96d7ed-1b43-45d9-9d84-64b9aaf7e694" (UID: "9e96d7ed-1b43-45d9-9d84-64b9aaf7e694"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:05.837379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.837320    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-tmp-dir\") pod \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") "
Oct 30 01:15:05.837379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.837375    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-tls-certs\") pod \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\" (UID: \"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694\") "
Oct 30 01:15:05.837603 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.837577    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-tmp-dir" pod "9e96d7ed-1b43-45d9-9d84-64b9aaf7e694" (UID: "9e96d7ed-1b43-45d9-9d84-64b9aaf7e694"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:05.838311 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.838289    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:05.838369 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.838320    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:05.838369 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.838335    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:05.839001 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.838978    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-kube-api-access-vrp5b" pod "9e96d7ed-1b43-45d9-9d84-64b9aaf7e694" (UID: "9e96d7ed-1b43-45d9-9d84-64b9aaf7e694"). InnerVolumeSpecName "kube-api-access-vrp5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:05.839176 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.839145    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-dshm" pod "9e96d7ed-1b43-45d9-9d84-64b9aaf7e694" (UID: "9e96d7ed-1b43-45d9-9d84-64b9aaf7e694"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:05.839370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.839341    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-tls-certs" pod "9e96d7ed-1b43-45d9-9d84-64b9aaf7e694" (UID: "9e96d7ed-1b43-45d9-9d84-64b9aaf7e694"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:15:05.939850 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.939821    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrp5b\" (UniqueName: \"kubernetes.io/projected/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-kube-api-access-vrp5b\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:05.939850 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.939847    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:05.939850 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:05.939857    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.646801 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:06.646757    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2" podUID="9e96d7ed-1b43-45d9-9d84-64b9aaf7e694" containerName="main" probeResult="failure" output="Get \"https://10.134.1.24:8000/ready\": context deadline exceeded"
Oct 30 01:15:06.669224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:06.669191    6099 scope.go:122] "RemoveContainer" containerID="e61d4b7eace3841a34adc5f2528c7fe23e290cf7e8a6bba6c8c46ebd80d2e4ca"
Oct 30 01:15:06.669494 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:06.669467    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg_odh-ai-gateway-infra(0b9a2818-c031-4bae-b1a2-d8793f88c00f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg" podUID="0b9a2818-c031-4bae-b1a2-d8793f88c00f"
Oct 30 01:15:06.670920 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:06.670892    6099 scope.go:122] "RemoveContainer" containerID="810383acd317e0cdd2c663c65a4fa34652dcd39cb5c6e00640c5e8246e7f841f"
Oct 30 01:15:06.671164 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:06.671142    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp_odh-ai-gateway-infra(9ffde68d-ddf9-439e-9cff-b0aab29c397e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp" podUID="9ffde68d-ddf9-439e-9cff-b0aab29c397e"
Oct 30 01:15:06.672553 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:06.672534    6099 scope.go:122] "RemoveContainer" containerID="ed6ecaff58545012c2585eb9a7badefd8b30fd6f1804fc17f9793beb1762ed89"
Oct 30 01:15:06.672830 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:06.672797    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl_odh-ai-gateway-infra(dc4a7cce-cbf8-43b2-99cd-e7a82ab54824)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl" podUID="dc4a7cce-cbf8-43b2-99cd-e7a82ab54824"
Oct 30 01:15:06.674368 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:06.674350    6099 scope.go:122] "RemoveContainer" containerID="f9dc67452dfda7a4c33e0c02516cd880c08dda9ccf818b94a207669aec4c9e30"
Oct 30 01:15:06.674603 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:06.674585    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d_odh-ai-gateway-infra(5cca91cc-d895-4aa2-b636-8d8dd97cdbb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d" podUID="5cca91cc-d895-4aa2-b636-8d8dd97cdbb8"
Oct 30 01:15:06.676424 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:06.676404    6099 scope.go:122] "RemoveContainer" containerID="0032769f338d729fd9ea09b5f755f1532c1f6361404006ab4c1d24fa1761f931"
Oct 30 01:15:06.676662 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:06.676640    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv_odh-ai-gateway-infra(b64df5bf-b9b0-47f9-bbd5-7c2839b42a53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv" podUID="b64df5bf-b9b0-47f9-bbd5-7c2839b42a53"
Oct 30 01:15:06.678035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:06.678017    6099 scope.go:122] "RemoveContainer" containerID="dda707756df08615c284922e27a30c699641e8c1b141975d9b4a3572758212b0"
Oct 30 01:15:06.678269 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:06.678247    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc_odh-ai-gateway-infra(7c9e6dd5-8118-494e-8cfd-bab30c938ddc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc" podUID="7c9e6dd5-8118-494e-8cfd-bab30c938ddc"
Oct 30 01:15:06.679719 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:06.679688    6099 scope.go:122] "RemoveContainer" containerID="01310b226242c0b3963db84f19c2ae6b25a73eaf85f688a5035c25d0bc1ea158"
Oct 30 01:15:06.679959 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:06.679935    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79_odh-ai-gateway-infra(2cf83038-adae-454f-81e4-5ae0e19c660b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79" podUID="2cf83038-adae-454f-81e4-5ae0e19c660b"
Oct 30 01:15:06.681386 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:06.681367    6099 scope.go:122] "RemoveContainer" containerID="0a7d134ffa34b710a5af79a4a03121437ee052f41c70ff3dbfb0788f78b701a3"
Oct 30 01:15:06.681623 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:06.681603    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw_odh-ai-gateway-infra(07c09d93-afed-4328-b7a4-23775614c03c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw" podUID="07c09d93-afed-4328-b7a4-23775614c03c"
Oct 30 01:15:06.683226 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:06.683206    6099 scope.go:122] "RemoveContainer" containerID="3edd280585a4c94c4ba1dcbdcf8da2c1ccb86e7de7cc2248e82a90c5bbb15947"
Oct 30 01:15:06.683509 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:06.683481    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr_odh-ai-gateway-infra(75974319-3673-4bdc-bb2c-e9be475a5723)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr" podUID="75974319-3673-4bdc-bb2c-e9be475a5723"
Oct 30 01:15:06.684683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:06.684662    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2" event={"ID":"9e96d7ed-1b43-45d9-9d84-64b9aaf7e694","Type":"ContainerDied","Data":"410f4c49163e2b93964692db08039aa8045a8eb40a210187f0c90722ce86be30"}
Oct 30 01:15:06.684810 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:06.684685    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"
Oct 30 01:15:06.684810 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:06.684718    6099 scope.go:122] "RemoveContainer" containerID="6ca883d51f797409e57035d6721e180878a99b5bd1b90e106acf32ca954cb302"
Oct 30 01:15:06.765589 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:06.765550    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"]
Oct 30 01:15:06.772474 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:06.772450    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-8e24a5/auth-test-model-8e24a5-kserve-5dcb65574f-f4kw2"]
Oct 30 01:15:06.809351 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:06.809329    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 01:15:06.827958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:06.827933    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 01:15:07.802877 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:07.802842    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"]
Oct 30 01:15:07.803315 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:07.803097    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr" podUID="80a38635-cb84-404c-9502-de7b43239b8e" containerName="istio-proxy" containerID="cri-o://71f282eb405fe0c8f715a5f2847c21d04a0bdad3bda5c881fbc614823dbc92cf" gracePeriod=30
Oct 30 01:15:08.329728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:08.327891    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e96d7ed-1b43-45d9-9d84-64b9aaf7e694" path="/var/lib/kubelet/pods/9e96d7ed-1b43-45d9-9d84-64b9aaf7e694/volumes"
Oct 30 01:15:09.430981 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:09.430944    6099 patch_prober.go:29] interesting pod/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:15:09.431445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:09.431008    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr" podUID="80a38635-cb84-404c-9502-de7b43239b8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:15:13.131579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.131546    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 01:15:13.219817 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.219781    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-workload-certs\") pod \"80a38635-cb84-404c-9502-de7b43239b8e\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") "
Oct 30 01:15:13.220025 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.219932    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a38635-cb84-404c-9502-de7b43239b8e-istio-token\" (UniqueName: \"kubernetes.io/projected/80a38635-cb84-404c-9502-de7b43239b8e-istio-token\") pod \"80a38635-cb84-404c-9502-de7b43239b8e\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") "
Oct 30 01:15:13.220210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.220179    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-workload-certs" pod "80a38635-cb84-404c-9502-de7b43239b8e" (UID: "80a38635-cb84-404c-9502-de7b43239b8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:13.220481 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.220457    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80a38635-cb84-404c-9502-de7b43239b8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a38635-cb84-404c-9502-de7b43239b8e-istio-podinfo\") pod \"80a38635-cb84-404c-9502-de7b43239b8e\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") "
Oct 30 01:15:13.220618 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.220534    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-credential-socket\") pod \"80a38635-cb84-404c-9502-de7b43239b8e\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") "
Oct 30 01:15:13.220618 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.220602    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-workload-socket\") pod \"80a38635-cb84-404c-9502-de7b43239b8e\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") "
Oct 30 01:15:13.220841 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.220684    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-istio-data\") pod \"80a38635-cb84-404c-9502-de7b43239b8e\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") "
Oct 30 01:15:13.220841 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.220749    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a38635-cb84-404c-9502-de7b43239b8e-kube-api-access-5k8bl\" (UniqueName: \"kubernetes.io/projected/80a38635-cb84-404c-9502-de7b43239b8e-kube-api-access-5k8bl\") pod \"80a38635-cb84-404c-9502-de7b43239b8e\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") "
Oct 30 01:15:13.220841 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.220805    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80a38635-cb84-404c-9502-de7b43239b8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a38635-cb84-404c-9502-de7b43239b8e-istiod-ca-cert\") pod \"80a38635-cb84-404c-9502-de7b43239b8e\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") "
Oct 30 01:15:13.221016 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.220857    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-istio-envoy\") pod \"80a38635-cb84-404c-9502-de7b43239b8e\" (UID: \"80a38635-cb84-404c-9502-de7b43239b8e\") "
Oct 30 01:15:13.221393 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.221367    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-workload-socket" pod "80a38635-cb84-404c-9502-de7b43239b8e" (UID: "80a38635-cb84-404c-9502-de7b43239b8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:13.221677 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.221623    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-istio-data" pod "80a38635-cb84-404c-9502-de7b43239b8e" (UID: "80a38635-cb84-404c-9502-de7b43239b8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:13.221912 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.221886    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a38635-cb84-404c-9502-de7b43239b8e-istiod-ca-cert" pod "80a38635-cb84-404c-9502-de7b43239b8e" (UID: "80a38635-cb84-404c-9502-de7b43239b8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:13.222039 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.221718    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-credential-socket" pod "80a38635-cb84-404c-9502-de7b43239b8e" (UID: "80a38635-cb84-404c-9502-de7b43239b8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:13.222880 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.222862    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:13.223003 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.222990    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:13.223090 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.223078    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:13.223172 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.223162    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a38635-cb84-404c-9502-de7b43239b8e-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:13.223245 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.223235    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:13.223891 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.223865    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a38635-cb84-404c-9502-de7b43239b8e-istio-token" pod "80a38635-cb84-404c-9502-de7b43239b8e" (UID: "80a38635-cb84-404c-9502-de7b43239b8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:13.224851 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.224825    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-istio-envoy" pod "80a38635-cb84-404c-9502-de7b43239b8e" (UID: "80a38635-cb84-404c-9502-de7b43239b8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:13.225823 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.225799    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80a38635-cb84-404c-9502-de7b43239b8e-istio-podinfo" pod "80a38635-cb84-404c-9502-de7b43239b8e" (UID: "80a38635-cb84-404c-9502-de7b43239b8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:15:13.226553 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.226524    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a38635-cb84-404c-9502-de7b43239b8e-kube-api-access-5k8bl" pod "80a38635-cb84-404c-9502-de7b43239b8e" (UID: "80a38635-cb84-404c-9502-de7b43239b8e"). InnerVolumeSpecName "kube-api-access-5k8bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:13.325683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.325650    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a38635-cb84-404c-9502-de7b43239b8e-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:13.325683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.325677    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a38635-cb84-404c-9502-de7b43239b8e-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:13.325683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.325685    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a38635-cb84-404c-9502-de7b43239b8e-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:13.325931 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.325694    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k8bl\" (UniqueName: \"kubernetes.io/projected/80a38635-cb84-404c-9502-de7b43239b8e-kube-api-access-5k8bl\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:13.475744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.475653    6099 scope.go:122] "RemoveContainer" containerID="50711919164e923856c921288125aeb8e9bb849946e47009b2207ad359b6f4ad"
Oct 30 01:15:13.488870 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.488845    6099 scope.go:122] "RemoveContainer" containerID="058160f1f3ac3f66a9d4c4a331493f0e048c88c5900dbe5931f60bfc0fbb96db"
Oct 30 01:15:13.502538 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.502519    6099 scope.go:122] "RemoveContainer" containerID="f00b518c3e4126a78dbd468479f84e385d5a4656ca492dda7dc85397350e3342"
Oct 30 01:15:13.515651 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.515574    6099 scope.go:122] "RemoveContainer" containerID="71f282eb405fe0c8f715a5f2847c21d04a0bdad3bda5c881fbc614823dbc92cf"
Oct 30 01:15:13.532309 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.532286    6099 scope.go:122] "RemoveContainer" containerID="5142a8d23d62dcd6141bc1ec047dc4dff125320b58f18323d0c59069436f6d1f"
Oct 30 01:15:13.547642 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.547622    6099 scope.go:122] "RemoveContainer" containerID="8956bb3f5adf2a646b14845bfb37d451f2db217e49b3cb7a696b064882f1864a"
Oct 30 01:15:13.562812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.559877    6099 scope.go:122] "RemoveContainer" containerID="d2c3cb89bc8ba7db78deeacf854449c57a587de059cbd016f4ebe311caa17faa"
Oct 30 01:15:13.604404 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.602682    6099 scope.go:122] "RemoveContainer" containerID="ed4056670b3ebe61eedbe6c3256947fcced9dcfe7abd89e769efd0046e189881"
Oct 30 01:15:13.617939 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.617913    6099 scope.go:122] "RemoveContainer" containerID="06fb38e8f1617638b68530c2e8eaf1f9cf0437637bb53c09b621bc3e9394a09c"
Oct 30 01:15:13.629329 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.629312    6099 scope.go:122] "RemoveContainer" containerID="a3d7ee417a824495fdd5b5f0386919c6b3d572dc1a4d8c78bc9b541ddd601bb0"
Oct 30 01:15:13.799771 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.799734    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr" event={"ID":"80a38635-cb84-404c-9502-de7b43239b8e","Type":"ContainerDied","Data":"71f282eb405fe0c8f715a5f2847c21d04a0bdad3bda5c881fbc614823dbc92cf"}
Oct 30 01:15:13.799906 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.799800    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr" event={"ID":"80a38635-cb84-404c-9502-de7b43239b8e","Type":"ContainerDied","Data":"524d3028bef5ce65bcf3e7bc880b8f6d6522892b804b0e0b914349ce438b63b7"}
Oct 30 01:15:13.799966 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.799929    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"
Oct 30 01:15:13.899310 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.899277    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"]
Oct 30 01:15:13.907545 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:13.907516    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-40ac41-openshift-default-5c6c78d655-xgdhr"]
Oct 30 01:15:14.265213 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:14.265183    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80a38635-cb84-404c-9502-de7b43239b8e" path="/var/lib/kubelet/pods/80a38635-cb84-404c-9502-de7b43239b8e/volumes"
Oct 30 01:15:16.260640 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:16.260611    6099 scope.go:122] "RemoveContainer" containerID="e61d4b7eace3841a34adc5f2528c7fe23e290cf7e8a6bba6c8c46ebd80d2e4ca"
Oct 30 01:15:16.261047 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:16.260815    6099 scope.go:122] "RemoveContainer" containerID="3edd280585a4c94c4ba1dcbdcf8da2c1ccb86e7de7cc2248e82a90c5bbb15947"
Oct 30 01:15:16.283054 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:16.283023    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j"]
Oct 30 01:15:16.632675 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:16.632637    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:15:16.842423 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:16.842337    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr" event={"ID":"75974319-3673-4bdc-bb2c-e9be475a5723","Type":"ContainerStarted","Data":"5da05c7acec571affc40ac74c6c2e77a0ddbcf86c7c68537f92fa31aa8c91743"}
Oct 30 01:15:17.254315 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:17.254238    6099 scope.go:122] "RemoveContainer" containerID="dda707756df08615c284922e27a30c699641e8c1b141975d9b4a3572758212b0"
Oct 30 01:15:17.254474 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:17.254394    6099 scope.go:122] "RemoveContainer" containerID="0032769f338d729fd9ea09b5f755f1532c1f6361404006ab4c1d24fa1761f931"
Oct 30 01:15:17.859928 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:17.859878    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv" event={"ID":"b64df5bf-b9b0-47f9-bbd5-7c2839b42a53","Type":"ContainerStarted","Data":"046b66bc3da71583e7087504aad1689ced5ad82072419bca99b99429f22c04c3"}
Oct 30 01:15:17.862350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:17.862318    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc" event={"ID":"7c9e6dd5-8118-494e-8cfd-bab30c938ddc","Type":"ContainerStarted","Data":"8867aaf334b00a642fd4c676a9cd1c244952abe434ba0aab8dd5e17f7f17f4e0"}
Oct 30 01:15:17.866415 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:17.866389    6099 generic.go:356] "Generic (PLEG): container finished" podID="75974319-3673-4bdc-bb2c-e9be475a5723" containerID="5da05c7acec571affc40ac74c6c2e77a0ddbcf86c7c68537f92fa31aa8c91743" exitCode=60
Oct 30 01:15:17.866530 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:17.866420    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr" event={"ID":"75974319-3673-4bdc-bb2c-e9be475a5723","Type":"ContainerDied","Data":"5da05c7acec571affc40ac74c6c2e77a0ddbcf86c7c68537f92fa31aa8c91743"}
Oct 30 01:15:17.866530 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:17.866483    6099 scope.go:122] "RemoveContainer" containerID="3edd280585a4c94c4ba1dcbdcf8da2c1ccb86e7de7cc2248e82a90c5bbb15947"
Oct 30 01:15:17.866982 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:17.866926    6099 scope.go:122] "RemoveContainer" containerID="5da05c7acec571affc40ac74c6c2e77a0ddbcf86c7c68537f92fa31aa8c91743"
Oct 30 01:15:17.867351 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:17.867327    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr_odh-ai-gateway-infra(75974319-3673-4bdc-bb2c-e9be475a5723)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr" podUID="75974319-3673-4bdc-bb2c-e9be475a5723"
Oct 30 01:15:17.869657 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:17.869637    6099 generic.go:356] "Generic (PLEG): container finished" podID="0b9a2818-c031-4bae-b1a2-d8793f88c00f" containerID="921e249604ef13f59b90e915a7ab53b456af2d917ce24e24b405d939efb38f35" exitCode=60
Oct 30 01:15:17.869774 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:17.869742    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg" event={"ID":"0b9a2818-c031-4bae-b1a2-d8793f88c00f","Type":"ContainerDied","Data":"921e249604ef13f59b90e915a7ab53b456af2d917ce24e24b405d939efb38f35"}
Oct 30 01:15:17.870176 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:17.870156    6099 scope.go:122] "RemoveContainer" containerID="921e249604ef13f59b90e915a7ab53b456af2d917ce24e24b405d939efb38f35"
Oct 30 01:15:17.870575 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:17.870552    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg_odh-ai-gateway-infra(0b9a2818-c031-4bae-b1a2-d8793f88c00f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg" podUID="0b9a2818-c031-4bae-b1a2-d8793f88c00f"
Oct 30 01:15:17.887202 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:17.887183    6099 scope.go:122] "RemoveContainer" containerID="e61d4b7eace3841a34adc5f2528c7fe23e290cf7e8a6bba6c8c46ebd80d2e4ca"
Oct 30 01:15:17.896946 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:17.896923    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr"]
Oct 30 01:15:18.255062 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:18.254943    6099 scope.go:122] "RemoveContainer" containerID="810383acd317e0cdd2c663c65a4fa34652dcd39cb5c6e00640c5e8246e7f841f"
Oct 30 01:15:18.883076 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:18.883035    6099 generic.go:356] "Generic (PLEG): container finished" podID="b64df5bf-b9b0-47f9-bbd5-7c2839b42a53" containerID="046b66bc3da71583e7087504aad1689ced5ad82072419bca99b99429f22c04c3" exitCode=60
Oct 30 01:15:18.883076 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:18.883065    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv" event={"ID":"b64df5bf-b9b0-47f9-bbd5-7c2839b42a53","Type":"ContainerDied","Data":"046b66bc3da71583e7087504aad1689ced5ad82072419bca99b99429f22c04c3"}
Oct 30 01:15:18.883548 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:18.883146    6099 scope.go:122] "RemoveContainer" containerID="0032769f338d729fd9ea09b5f755f1532c1f6361404006ab4c1d24fa1761f931"
Oct 30 01:15:18.883795 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:18.883759    6099 scope.go:122] "RemoveContainer" containerID="046b66bc3da71583e7087504aad1689ced5ad82072419bca99b99429f22c04c3"
Oct 30 01:15:18.884347 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:18.884259    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv_odh-ai-gateway-infra(b64df5bf-b9b0-47f9-bbd5-7c2839b42a53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv" podUID="b64df5bf-b9b0-47f9-bbd5-7c2839b42a53"
Oct 30 01:15:18.885701 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:18.885678    6099 generic.go:356] "Generic (PLEG): container finished" podID="7c9e6dd5-8118-494e-8cfd-bab30c938ddc" containerID="8867aaf334b00a642fd4c676a9cd1c244952abe434ba0aab8dd5e17f7f17f4e0" exitCode=60
Oct 30 01:15:18.885868 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:18.885767    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc" event={"ID":"7c9e6dd5-8118-494e-8cfd-bab30c938ddc","Type":"ContainerDied","Data":"8867aaf334b00a642fd4c676a9cd1c244952abe434ba0aab8dd5e17f7f17f4e0"}
Oct 30 01:15:18.886243 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:18.886226    6099 scope.go:122] "RemoveContainer" containerID="8867aaf334b00a642fd4c676a9cd1c244952abe434ba0aab8dd5e17f7f17f4e0"
Oct 30 01:15:18.886655 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:18.886634    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc_odh-ai-gateway-infra(7c9e6dd5-8118-494e-8cfd-bab30c938ddc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc" podUID="7c9e6dd5-8118-494e-8cfd-bab30c938ddc"
Oct 30 01:15:18.892278 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:18.892253    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp" event={"ID":"9ffde68d-ddf9-439e-9cff-b0aab29c397e","Type":"ContainerStarted","Data":"004e8c53da1a241982a4d7b978dd9507dec4f37478f3233de61557148512bd38"}
Oct 30 01:15:18.899282 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:18.899261    6099 scope.go:122] "RemoveContainer" containerID="dda707756df08615c284922e27a30c699641e8c1b141975d9b4a3572758212b0"
Oct 30 01:15:18.926342 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:18.926315    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv"]
Oct 30 01:15:19.017795 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:19.017700    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg"]
Oct 30 01:15:19.043057 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:19.042805    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc"]
Oct 30 01:15:19.273731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:19.262637    6099 scope.go:122] "RemoveContainer" containerID="0a7d134ffa34b710a5af79a4a03121437ee052f41c70ff3dbfb0788f78b701a3"
Oct 30 01:15:19.273731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:19.263776    6099 scope.go:122] "RemoveContainer" containerID="f9dc67452dfda7a4c33e0c02516cd880c08dda9ccf818b94a207669aec4c9e30"
Oct 30 01:15:19.273731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:19.264593    6099 scope.go:122] "RemoveContainer" containerID="01310b226242c0b3963db84f19c2ae6b25a73eaf85f688a5035c25d0bc1ea158"
Oct 30 01:15:19.273731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:19.265328    6099 scope.go:122] "RemoveContainer" containerID="ed6ecaff58545012c2585eb9a7badefd8b30fd6f1804fc17f9793beb1762ed89"
Oct 30 01:15:19.602112 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:19.597910    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr"
Oct 30 01:15:19.788730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:19.765459    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75974319-3673-4bdc-bb2c-e9be475a5723-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75974319-3673-4bdc-bb2c-e9be475a5723-maas-api-service-ca\") pod \"75974319-3673-4bdc-bb2c-e9be475a5723\" (UID: \"75974319-3673-4bdc-bb2c-e9be475a5723\") "
Oct 30 01:15:19.789125 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:19.789100    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75974319-3673-4bdc-bb2c-e9be475a5723-kube-api-access-6krwp\" (UniqueName: \"kubernetes.io/projected/75974319-3673-4bdc-bb2c-e9be475a5723-kube-api-access-6krwp\") pod \"75974319-3673-4bdc-bb2c-e9be475a5723\" (UID: \"75974319-3673-4bdc-bb2c-e9be475a5723\") "
Oct 30 01:15:19.790667 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:19.777466    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75974319-3673-4bdc-bb2c-e9be475a5723-maas-api-service-ca" pod "75974319-3673-4bdc-bb2c-e9be475a5723" (UID: "75974319-3673-4bdc-bb2c-e9be475a5723"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:19.895731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:19.853639    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75974319-3673-4bdc-bb2c-e9be475a5723-kube-api-access-6krwp" pod "75974319-3673-4bdc-bb2c-e9be475a5723" (UID: "75974319-3673-4bdc-bb2c-e9be475a5723"). InnerVolumeSpecName "kube-api-access-6krwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:19.907653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:19.907615    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6krwp\" (UniqueName: \"kubernetes.io/projected/75974319-3673-4bdc-bb2c-e9be475a5723-kube-api-access-6krwp\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:19.908305 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:19.908283    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75974319-3673-4bdc-bb2c-e9be475a5723-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.096995 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:20.096891    6099 generic.go:356] "Generic (PLEG): container finished" podID="9ffde68d-ddf9-439e-9cff-b0aab29c397e" containerID="004e8c53da1a241982a4d7b978dd9507dec4f37478f3233de61557148512bd38" exitCode=60
Oct 30 01:15:20.126391 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:20.126280    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp" event={"ID":"9ffde68d-ddf9-439e-9cff-b0aab29c397e","Type":"ContainerDied","Data":"004e8c53da1a241982a4d7b978dd9507dec4f37478f3233de61557148512bd38"}
Oct 30 01:15:20.126579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:20.126421    6099 scope.go:122] "RemoveContainer" containerID="810383acd317e0cdd2c663c65a4fa34652dcd39cb5c6e00640c5e8246e7f841f"
Oct 30 01:15:20.159544 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:20.127156    6099 scope.go:122] "RemoveContainer" containerID="004e8c53da1a241982a4d7b978dd9507dec4f37478f3233de61557148512bd38"
Oct 30 01:15:20.173727 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:20.172599    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp_odh-ai-gateway-infra(9ffde68d-ddf9-439e-9cff-b0aab29c397e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp" podUID="9ffde68d-ddf9-439e-9cff-b0aab29c397e"
Oct 30 01:15:20.247946 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:20.244867    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg"
Oct 30 01:15:20.365165 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:20.349695    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b9a2818-c031-4bae-b1a2-d8793f88c00f-kube-api-access-4tb2f\" (UniqueName: \"kubernetes.io/projected/0b9a2818-c031-4bae-b1a2-d8793f88c00f-kube-api-access-4tb2f\") pod \"0b9a2818-c031-4bae-b1a2-d8793f88c00f\" (UID: \"0b9a2818-c031-4bae-b1a2-d8793f88c00f\") "
Oct 30 01:15:20.365165 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:20.349886    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b9a2818-c031-4bae-b1a2-d8793f88c00f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9a2818-c031-4bae-b1a2-d8793f88c00f-maas-api-service-ca\") pod \"0b9a2818-c031-4bae-b1a2-d8793f88c00f\" (UID: \"0b9a2818-c031-4bae-b1a2-d8793f88c00f\") "
Oct 30 01:15:20.406733 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:20.370008    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b9a2818-c031-4bae-b1a2-d8793f88c00f-maas-api-service-ca" pod "0b9a2818-c031-4bae-b1a2-d8793f88c00f" (UID: "0b9a2818-c031-4bae-b1a2-d8793f88c00f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:20.502743 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:20.470514    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9a2818-c031-4bae-b1a2-d8793f88c00f-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.516856 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:20.515942    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp"]
Oct 30 01:15:20.567798 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:20.561910    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b9a2818-c031-4bae-b1a2-d8793f88c00f-kube-api-access-4tb2f" pod "0b9a2818-c031-4bae-b1a2-d8793f88c00f" (UID: "0b9a2818-c031-4bae-b1a2-d8793f88c00f"). InnerVolumeSpecName "kube-api-access-4tb2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:20.567798 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:20.562627    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr"
Oct 30 01:15:20.581852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:20.572566    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tb2f\" (UniqueName: \"kubernetes.io/projected/0b9a2818-c031-4bae-b1a2-d8793f88c00f-kube-api-access-4tb2f\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.582204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:20.582162    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr" event={"ID":"75974319-3673-4bdc-bb2c-e9be475a5723","Type":"ContainerDied","Data":"3b6030897464383cf55629bcbcc58ff613282241d4dad12e2fd04fce562b305b"}
Oct 30 01:15:20.582524 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:20.582454    6099 scope.go:122] "RemoveContainer" containerID="5da05c7acec571affc40ac74c6c2e77a0ddbcf86c7c68537f92fa31aa8c91743"
Oct 30 01:15:21.060952 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:21.060860    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr"]
Oct 30 01:15:21.060952 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:21.060912    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae06fe-29888715-dn9cr"]
Oct 30 01:15:21.088732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:21.079040    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26" podUID="6b163830-1b90-4bcc-8238-a08bc5b9cb7e" containerName="main" probeResult="failure" output="Get \"https://10.134.1.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:15:21.284928 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:21.229841    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 30 01:15:21.284928 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:21.263449    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn" podUID="e36adae7-03ed-4647-9410-5b4bceafa593" containerName="main" probeResult="failure" output="Get \"https://10.134.1.17:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:15:21.316761 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:21.285434    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j" podUID="40f9d853-515b-48e7-b72b-5fd68d43ac4f" containerName="payload-processing" containerID="cri-o://50357b7fb72e57c1c330b222664b98ea5490e8ab14551d2d4b1a9d9097d11f9e" gracePeriod=25
Oct 30 01:15:21.316761 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:21.294793    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:15:21.371754 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:21.368199    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:15:21.445750 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:21.427834    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:15:21.477865 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:21.456118    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:15:21.477865 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:21.456811    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:15:21.477865 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:21.458241    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:15:21.477865 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:21.472353    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 01:15:21.544885 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:21.530261    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:15:21.560807 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:21.559762    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 01:15:21.613263 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:21.562520    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:15:21.962674 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:21.962587    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26" podUID="6b163830-1b90-4bcc-8238-a08bc5b9cb7e" containerName="main" probeResult="failure" output="Get \"https://10.134.1.26:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:15:22.072149 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.032822    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:15:22.089072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.086173    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26" podUID="6b163830-1b90-4bcc-8238-a08bc5b9cb7e" containerName="main" probeResult="failure" output="Get \"https://10.134.1.26:8000/ready\": context deadline exceeded"
Oct 30 01:15:22.089072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.087274    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:15:22.233780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.227819    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg" event={"ID":"0b9a2818-c031-4bae-b1a2-d8793f88c00f","Type":"ContainerDied","Data":"2ae525b62c4280eee763c591887b0b2784003ed82a05454e0741f795a9402ea7"}
Oct 30 01:15:22.233780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.227888    6099 scope.go:122] "RemoveContainer" containerID="921e249604ef13f59b90e915a7ab53b456af2d917ce24e24b405d939efb38f35"
Oct 30 01:15:22.233780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.228109    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg"
Oct 30 01:15:22.246899 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.246794    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:15:22.326993 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.324667    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75974319-3673-4bdc-bb2c-e9be475a5723" path="/var/lib/kubelet/pods/75974319-3673-4bdc-bb2c-e9be475a5723/volumes"
Oct 30 01:15:22.329686 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.329659    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d" event={"ID":"5cca91cc-d895-4aa2-b636-8d8dd97cdbb8","Type":"ContainerStarted","Data":"b88b2180e91e4f7cfca8f3b84a03ec2163f13b4ff8f16dd651806e37d94e442a"}
Oct 30 01:15:22.331452 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.331430    6099 scope.go:122] "RemoveContainer" containerID="b88b2180e91e4f7cfca8f3b84a03ec2163f13b4ff8f16dd651806e37d94e442a"
Oct 30 01:15:22.335476 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:22.332014    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d_odh-ai-gateway-infra(5cca91cc-d895-4aa2-b636-8d8dd97cdbb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d" podUID="5cca91cc-d895-4aa2-b636-8d8dd97cdbb8"
Oct 30 01:15:22.335641 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.333603    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79" event={"ID":"2cf83038-adae-454f-81e4-5ae0e19c660b","Type":"ContainerStarted","Data":"d6659e14d3dea18b1c4f06c021e2587a82b93f996e5b3f7882b5219487acca23"}
Oct 30 01:15:22.336258 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.336238    6099 scope.go:122] "RemoveContainer" containerID="d6659e14d3dea18b1c4f06c021e2587a82b93f996e5b3f7882b5219487acca23"
Oct 30 01:15:22.341558 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:22.341530    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79_odh-ai-gateway-infra(2cf83038-adae-454f-81e4-5ae0e19c660b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79" podUID="2cf83038-adae-454f-81e4-5ae0e19c660b"
Oct 30 01:15:22.659544 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.659489    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg"]
Oct 30 01:15:22.664564 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.664532    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-95492e-29888715-5vqlg"]
Oct 30 01:15:22.742057 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.742031    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv"
Oct 30 01:15:22.744626 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.744604    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc"
Oct 30 01:15:22.757969 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.757942    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9e6dd5-8118-494e-8cfd-bab30c938ddc-kube-api-access-fc7qb\" (UniqueName: \"kubernetes.io/projected/7c9e6dd5-8118-494e-8cfd-bab30c938ddc-kube-api-access-fc7qb\") pod \"7c9e6dd5-8118-494e-8cfd-bab30c938ddc\" (UID: \"7c9e6dd5-8118-494e-8cfd-bab30c938ddc\") "
Oct 30 01:15:22.758786 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.758557    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64df5bf-b9b0-47f9-bbd5-7c2839b42a53-kube-api-access-qn247\" (UniqueName: \"kubernetes.io/projected/b64df5bf-b9b0-47f9-bbd5-7c2839b42a53-kube-api-access-qn247\") pod \"b64df5bf-b9b0-47f9-bbd5-7c2839b42a53\" (UID: \"b64df5bf-b9b0-47f9-bbd5-7c2839b42a53\") "
Oct 30 01:15:22.758786 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.758646    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c9e6dd5-8118-494e-8cfd-bab30c938ddc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c9e6dd5-8118-494e-8cfd-bab30c938ddc-maas-api-service-ca\") pod \"7c9e6dd5-8118-494e-8cfd-bab30c938ddc\" (UID: \"7c9e6dd5-8118-494e-8cfd-bab30c938ddc\") "
Oct 30 01:15:22.758786 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.758740    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64df5bf-b9b0-47f9-bbd5-7c2839b42a53-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b64df5bf-b9b0-47f9-bbd5-7c2839b42a53-maas-api-service-ca\") pod \"b64df5bf-b9b0-47f9-bbd5-7c2839b42a53\" (UID: \"b64df5bf-b9b0-47f9-bbd5-7c2839b42a53\") "
Oct 30 01:15:22.759887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.759840    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9e6dd5-8118-494e-8cfd-bab30c938ddc-maas-api-service-ca" pod "7c9e6dd5-8118-494e-8cfd-bab30c938ddc" (UID: "7c9e6dd5-8118-494e-8cfd-bab30c938ddc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:22.760532 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.760211    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64df5bf-b9b0-47f9-bbd5-7c2839b42a53-maas-api-service-ca" pod "b64df5bf-b9b0-47f9-bbd5-7c2839b42a53" (UID: "b64df5bf-b9b0-47f9-bbd5-7c2839b42a53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:22.760532 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.760479    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c9e6dd5-8118-494e-8cfd-bab30c938ddc-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.760532 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.760509    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b64df5bf-b9b0-47f9-bbd5-7c2839b42a53-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.761470 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.761441    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9e6dd5-8118-494e-8cfd-bab30c938ddc-kube-api-access-fc7qb" pod "7c9e6dd5-8118-494e-8cfd-bab30c938ddc" (UID: "7c9e6dd5-8118-494e-8cfd-bab30c938ddc"). InnerVolumeSpecName "kube-api-access-fc7qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:22.763552 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.763527    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64df5bf-b9b0-47f9-bbd5-7c2839b42a53-kube-api-access-qn247" pod "b64df5bf-b9b0-47f9-bbd5-7c2839b42a53" (UID: "b64df5bf-b9b0-47f9-bbd5-7c2839b42a53"). InnerVolumeSpecName "kube-api-access-qn247". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:22.830493 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.830466    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp"
Oct 30 01:15:22.834300 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:22.834227    6099 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4a7cce_cbf8_43b2_99cd_e7a82ab54824.slice/crio-conmon-fab61c4b527c94dcbf29e282a4c274c718bd10bc394645a5d4fda41198534604.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9a2818_c031_4bae_b1a2_d8793f88c00f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c09d93_afed_4328_b7a4_23775614c03c.slice/crio-conmon-ca4453415a4e2612f5de98f8069c59faa657eda9dbb244a48073d0b9970d5cf0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4a7cce_cbf8_43b2_99cd_e7a82ab54824.slice/crio-fab61c4b527c94dcbf29e282a4c274c718bd10bc394645a5d4fda41198534604.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 01:15:22.862165 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.862130    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ffde68d-ddf9-439e-9cff-b0aab29c397e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffde68d-ddf9-439e-9cff-b0aab29c397e-maas-api-service-ca\") pod \"9ffde68d-ddf9-439e-9cff-b0aab29c397e\" (UID: \"9ffde68d-ddf9-439e-9cff-b0aab29c397e\") "
Oct 30 01:15:22.862620 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.862595    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ffde68d-ddf9-439e-9cff-b0aab29c397e-kube-api-access-vq5g2\" (UniqueName: \"kubernetes.io/projected/9ffde68d-ddf9-439e-9cff-b0aab29c397e-kube-api-access-vq5g2\") pod \"9ffde68d-ddf9-439e-9cff-b0aab29c397e\" (UID: \"9ffde68d-ddf9-439e-9cff-b0aab29c397e\") "
Oct 30 01:15:22.863535 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.863455    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ffde68d-ddf9-439e-9cff-b0aab29c397e-maas-api-service-ca" pod "9ffde68d-ddf9-439e-9cff-b0aab29c397e" (UID: "9ffde68d-ddf9-439e-9cff-b0aab29c397e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:22.864288 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.863596    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc7qb\" (UniqueName: \"kubernetes.io/projected/7c9e6dd5-8118-494e-8cfd-bab30c938ddc-kube-api-access-fc7qb\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.864437 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.864419    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn247\" (UniqueName: \"kubernetes.io/projected/b64df5bf-b9b0-47f9-bbd5-7c2839b42a53-kube-api-access-qn247\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.877882 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.877850    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ffde68d-ddf9-439e-9cff-b0aab29c397e-kube-api-access-vq5g2" pod "9ffde68d-ddf9-439e-9cff-b0aab29c397e" (UID: "9ffde68d-ddf9-439e-9cff-b0aab29c397e"). InnerVolumeSpecName "kube-api-access-vq5g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:22.958443 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.958413    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j"
Oct 30 01:15:23.019135 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:22.966129    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq5g2\" (UniqueName: \"kubernetes.io/projected/9ffde68d-ddf9-439e-9cff-b0aab29c397e-kube-api-access-vq5g2\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:23.019135 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.018833    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffde68d-ddf9-439e-9cff-b0aab29c397e-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:23.120063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.119985    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f9d853-515b-48e7-b72b-5fd68d43ac4f-kube-api-access-67wlb\" (UniqueName: \"kubernetes.io/projected/40f9d853-515b-48e7-b72b-5fd68d43ac4f-kube-api-access-67wlb\") pod \"40f9d853-515b-48e7-b72b-5fd68d43ac4f\" (UID: \"40f9d853-515b-48e7-b72b-5fd68d43ac4f\") "
Oct 30 01:15:23.120528 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.120098    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f9d853-515b-48e7-b72b-5fd68d43ac4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f9d853-515b-48e7-b72b-5fd68d43ac4f-plugins-config-volume\") pod \"40f9d853-515b-48e7-b72b-5fd68d43ac4f\" (UID: \"40f9d853-515b-48e7-b72b-5fd68d43ac4f\") "
Oct 30 01:15:23.120608 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.120528    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f9d853-515b-48e7-b72b-5fd68d43ac4f-plugins-config-volume" pod "40f9d853-515b-48e7-b72b-5fd68d43ac4f" (UID: "40f9d853-515b-48e7-b72b-5fd68d43ac4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:23.120948 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.120927    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f9d853-515b-48e7-b72b-5fd68d43ac4f-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:23.122145 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.122126    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f9d853-515b-48e7-b72b-5fd68d43ac4f-kube-api-access-67wlb" pod "40f9d853-515b-48e7-b72b-5fd68d43ac4f" (UID: "40f9d853-515b-48e7-b72b-5fd68d43ac4f"). InnerVolumeSpecName "kube-api-access-67wlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:23.222739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.222691    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67wlb\" (UniqueName: \"kubernetes.io/projected/40f9d853-515b-48e7-b72b-5fd68d43ac4f-kube-api-access-67wlb\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:23.395614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.395528    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79"]
Oct 30 01:15:23.398817 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.398793    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp"
Oct 30 01:15:23.398817 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.398803    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp" event={"ID":"9ffde68d-ddf9-439e-9cff-b0aab29c397e","Type":"ContainerDied","Data":"7ca1948ff2cee6aa501935f869f5e08d3f926bfa03b76e3f171ad67af98c4291"}
Oct 30 01:15:23.399028 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.398862    6099 scope.go:122] "RemoveContainer" containerID="004e8c53da1a241982a4d7b978dd9507dec4f37478f3233de61557148512bd38"
Oct 30 01:15:23.401099 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.401074    6099 generic.go:356] "Generic (PLEG): container finished" podID="dc4a7cce-cbf8-43b2-99cd-e7a82ab54824" containerID="fab61c4b527c94dcbf29e282a4c274c718bd10bc394645a5d4fda41198534604" exitCode=60
Oct 30 01:15:23.401242 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.401142    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl" event={"ID":"dc4a7cce-cbf8-43b2-99cd-e7a82ab54824","Type":"ContainerDied","Data":"fab61c4b527c94dcbf29e282a4c274c718bd10bc394645a5d4fda41198534604"}
Oct 30 01:15:23.401639 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.401617    6099 scope.go:122] "RemoveContainer" containerID="fab61c4b527c94dcbf29e282a4c274c718bd10bc394645a5d4fda41198534604"
Oct 30 01:15:23.402200 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:23.402172    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl_odh-ai-gateway-infra(dc4a7cce-cbf8-43b2-99cd-e7a82ab54824)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl" podUID="dc4a7cce-cbf8-43b2-99cd-e7a82ab54824"
Oct 30 01:15:23.403676 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.403644    6099 generic.go:356] "Generic (PLEG): container finished" podID="5cca91cc-d895-4aa2-b636-8d8dd97cdbb8" containerID="b88b2180e91e4f7cfca8f3b84a03ec2163f13b4ff8f16dd651806e37d94e442a" exitCode=60
Oct 30 01:15:23.403834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.403682    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d" event={"ID":"5cca91cc-d895-4aa2-b636-8d8dd97cdbb8","Type":"ContainerDied","Data":"b88b2180e91e4f7cfca8f3b84a03ec2163f13b4ff8f16dd651806e37d94e442a"}
Oct 30 01:15:23.404146 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.404125    6099 scope.go:122] "RemoveContainer" containerID="b88b2180e91e4f7cfca8f3b84a03ec2163f13b4ff8f16dd651806e37d94e442a"
Oct 30 01:15:23.404448 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:23.404428    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d_odh-ai-gateway-infra(5cca91cc-d895-4aa2-b636-8d8dd97cdbb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d" podUID="5cca91cc-d895-4aa2-b636-8d8dd97cdbb8"
Oct 30 01:15:23.405772 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.405751    6099 generic.go:356] "Generic (PLEG): container finished" podID="40f9d853-515b-48e7-b72b-5fd68d43ac4f" containerID="50357b7fb72e57c1c330b222664b98ea5490e8ab14551d2d4b1a9d9097d11f9e" exitCode=0
Oct 30 01:15:23.405890 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.405790    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j" event={"ID":"40f9d853-515b-48e7-b72b-5fd68d43ac4f","Type":"ContainerDied","Data":"50357b7fb72e57c1c330b222664b98ea5490e8ab14551d2d4b1a9d9097d11f9e"}
Oct 30 01:15:23.405890 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.405838    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j" event={"ID":"40f9d853-515b-48e7-b72b-5fd68d43ac4f","Type":"ContainerDied","Data":"bbfd2d6bfc45320ee93e72563e681c6a2e9518ee2220972f8ca50e60efe0a783"}
Oct 30 01:15:23.405890 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.405802    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j"
Oct 30 01:15:23.407227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.407209    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv"
Oct 30 01:15:23.407318 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.407265    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv" event={"ID":"b64df5bf-b9b0-47f9-bbd5-7c2839b42a53","Type":"ContainerDied","Data":"af0b0a5685ec99b0f6ddb2452ab70bb0955ad7a96d9522d35c850245e45d20ef"}
Oct 30 01:15:23.413845 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.413825    6099 scope.go:122] "RemoveContainer" containerID="ed6ecaff58545012c2585eb9a7badefd8b30fd6f1804fc17f9793beb1762ed89"
Oct 30 01:15:23.415577 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.415561    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc"
Oct 30 01:15:23.415864 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.415581    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc" event={"ID":"7c9e6dd5-8118-494e-8cfd-bab30c938ddc","Type":"ContainerDied","Data":"e12bc6ab53e2ff994c26d546d688415df4518d53282d1edacfea441d16afb533"}
Oct 30 01:15:23.419148 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.419129    6099 generic.go:356] "Generic (PLEG): container finished" podID="2cf83038-adae-454f-81e4-5ae0e19c660b" containerID="d6659e14d3dea18b1c4f06c021e2587a82b93f996e5b3f7882b5219487acca23" exitCode=60
Oct 30 01:15:23.419339 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.419311    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79" event={"ID":"2cf83038-adae-454f-81e4-5ae0e19c660b","Type":"ContainerDied","Data":"d6659e14d3dea18b1c4f06c021e2587a82b93f996e5b3f7882b5219487acca23"}
Oct 30 01:15:23.424994 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.424972    6099 generic.go:356] "Generic (PLEG): container finished" podID="07c09d93-afed-4328-b7a4-23775614c03c" containerID="ca4453415a4e2612f5de98f8069c59faa657eda9dbb244a48073d0b9970d5cf0" exitCode=60
Oct 30 01:15:23.425179 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.425153    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw" event={"ID":"07c09d93-afed-4328-b7a4-23775614c03c","Type":"ContainerDied","Data":"ca4453415a4e2612f5de98f8069c59faa657eda9dbb244a48073d0b9970d5cf0"}
Oct 30 01:15:23.425629 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.425614    6099 scope.go:122] "RemoveContainer" containerID="ca4453415a4e2612f5de98f8069c59faa657eda9dbb244a48073d0b9970d5cf0"
Oct 30 01:15:23.426212 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:23.426121    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw_odh-ai-gateway-infra(07c09d93-afed-4328-b7a4-23775614c03c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw" podUID="07c09d93-afed-4328-b7a4-23775614c03c"
Oct 30 01:15:23.432115 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.432097    6099 scope.go:122] "RemoveContainer" containerID="f9dc67452dfda7a4c33e0c02516cd880c08dda9ccf818b94a207669aec4c9e30"
Oct 30 01:15:23.452673 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.452037    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d"]
Oct 30 01:15:23.460125 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.460093    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp"]
Oct 30 01:15:23.475363 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.471851    6099 scope.go:122] "RemoveContainer" containerID="50357b7fb72e57c1c330b222664b98ea5490e8ab14551d2d4b1a9d9097d11f9e"
Oct 30 01:15:23.493601 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.491314    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-16c1bb-29888715-74bnp"]
Oct 30 01:15:23.606208 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.606069    6099 scope.go:122] "RemoveContainer" containerID="50357b7fb72e57c1c330b222664b98ea5490e8ab14551d2d4b1a9d9097d11f9e"
Oct 30 01:15:23.606621 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:23.606514    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50357b7fb72e57c1c330b222664b98ea5490e8ab14551d2d4b1a9d9097d11f9e\": container with ID starting with 50357b7fb72e57c1c330b222664b98ea5490e8ab14551d2d4b1a9d9097d11f9e not found: ID does not exist" containerID="50357b7fb72e57c1c330b222664b98ea5490e8ab14551d2d4b1a9d9097d11f9e"
Oct 30 01:15:23.606621 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.606572    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50357b7fb72e57c1c330b222664b98ea5490e8ab14551d2d4b1a9d9097d11f9e"} err="failed to get container status \"50357b7fb72e57c1c330b222664b98ea5490e8ab14551d2d4b1a9d9097d11f9e\": rpc error: code = NotFound desc = could not find container \"50357b7fb72e57c1c330b222664b98ea5490e8ab14551d2d4b1a9d9097d11f9e\": container with ID starting with 50357b7fb72e57c1c330b222664b98ea5490e8ab14551d2d4b1a9d9097d11f9e not found: ID does not exist"
Oct 30 01:15:23.606928 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.606599    6099 scope.go:122] "RemoveContainer" containerID="046b66bc3da71583e7087504aad1689ced5ad82072419bca99b99429f22c04c3"
Oct 30 01:15:23.621118 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.621095    6099 scope.go:122] "RemoveContainer" containerID="8867aaf334b00a642fd4c676a9cd1c244952abe434ba0aab8dd5e17f7f17f4e0"
Oct 30 01:15:23.637235 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.637214    6099 scope.go:122] "RemoveContainer" containerID="01310b226242c0b3963db84f19c2ae6b25a73eaf85f688a5035c25d0bc1ea158"
Oct 30 01:15:23.698847 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.698688    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j"]
Oct 30 01:15:23.704929 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.704875    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91e87-cc47bfbdd-qzn6j"]
Oct 30 01:15:23.705384 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.705331    6099 scope.go:122] "RemoveContainer" containerID="0a7d134ffa34b710a5af79a4a03121437ee052f41c70ff3dbfb0788f78b701a3"
Oct 30 01:15:23.749368 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.749343    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79"
Oct 30 01:15:23.823116 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.823080    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc"]
Oct 30 01:15:23.831515 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.831473    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5062f6-29888715-jdkhc"]
Oct 30 01:15:23.836447 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.836413    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf83038-adae-454f-81e4-5ae0e19c660b-kube-api-access-t4hz2\" (UniqueName: \"kubernetes.io/projected/2cf83038-adae-454f-81e4-5ae0e19c660b-kube-api-access-t4hz2\") pod \"2cf83038-adae-454f-81e4-5ae0e19c660b\" (UID: \"2cf83038-adae-454f-81e4-5ae0e19c660b\") "
Oct 30 01:15:23.837204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.836823    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf83038-adae-454f-81e4-5ae0e19c660b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf83038-adae-454f-81e4-5ae0e19c660b-maas-api-service-ca\") pod \"2cf83038-adae-454f-81e4-5ae0e19c660b\" (UID: \"2cf83038-adae-454f-81e4-5ae0e19c660b\") "
Oct 30 01:15:23.837204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.837161    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf83038-adae-454f-81e4-5ae0e19c660b-maas-api-service-ca" pod "2cf83038-adae-454f-81e4-5ae0e19c660b" (UID: "2cf83038-adae-454f-81e4-5ae0e19c660b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:23.838072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.838039    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf83038-adae-454f-81e4-5ae0e19c660b-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:23.851258 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.851226    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf83038-adae-454f-81e4-5ae0e19c660b-kube-api-access-t4hz2" pod "2cf83038-adae-454f-81e4-5ae0e19c660b" (UID: "2cf83038-adae-454f-81e4-5ae0e19c660b"). InnerVolumeSpecName "kube-api-access-t4hz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:23.934791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.934755    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv"]
Oct 30 01:15:23.943767 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.939773    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bea3ef-29888715-6ffgv"]
Oct 30 01:15:23.944046 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:23.944005    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4hz2\" (UniqueName: \"kubernetes.io/projected/2cf83038-adae-454f-81e4-5ae0e19c660b-kube-api-access-t4hz2\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:24.291063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:24.291023    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b9a2818-c031-4bae-b1a2-d8793f88c00f" path="/var/lib/kubelet/pods/0b9a2818-c031-4bae-b1a2-d8793f88c00f/volumes"
Oct 30 01:15:24.291802 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:24.291742    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f9d853-515b-48e7-b72b-5fd68d43ac4f" path="/var/lib/kubelet/pods/40f9d853-515b-48e7-b72b-5fd68d43ac4f/volumes"
Oct 30 01:15:24.292248 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:24.292227    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c9e6dd5-8118-494e-8cfd-bab30c938ddc" path="/var/lib/kubelet/pods/7c9e6dd5-8118-494e-8cfd-bab30c938ddc/volumes"
Oct 30 01:15:24.293109 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:24.293086    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ffde68d-ddf9-439e-9cff-b0aab29c397e" path="/var/lib/kubelet/pods/9ffde68d-ddf9-439e-9cff-b0aab29c397e/volumes"
Oct 30 01:15:24.293576 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:24.293555    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64df5bf-b9b0-47f9-bbd5-7c2839b42a53" path="/var/lib/kubelet/pods/b64df5bf-b9b0-47f9-bbd5-7c2839b42a53/volumes"
Oct 30 01:15:24.451908 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:24.451866    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79"
Oct 30 01:15:24.452049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:24.451896    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79" event={"ID":"2cf83038-adae-454f-81e4-5ae0e19c660b","Type":"ContainerDied","Data":"8838bf645cb96c50bc8ada0bbc8c3c6bda5c393739ea37cb74cbdc20fad1765a"}
Oct 30 01:15:24.452049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:24.452034    6099 scope.go:122] "RemoveContainer" containerID="d6659e14d3dea18b1c4f06c021e2587a82b93f996e5b3f7882b5219487acca23"
Oct 30 01:15:24.538847 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:24.538811    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79"]
Oct 30 01:15:24.544826 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:24.543834    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b3593a-29888715-45l79"]
Oct 30 01:15:24.638585 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:24.638533    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl"]
Oct 30 01:15:24.749078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:24.749048    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw"]
Oct 30 01:15:24.807812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:24.807737    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d"
Oct 30 01:15:24.888798 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:24.888759    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cca91cc-d895-4aa2-b636-8d8dd97cdbb8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cca91cc-d895-4aa2-b636-8d8dd97cdbb8-maas-api-service-ca\") pod \"5cca91cc-d895-4aa2-b636-8d8dd97cdbb8\" (UID: \"5cca91cc-d895-4aa2-b636-8d8dd97cdbb8\") "
Oct 30 01:15:24.889194 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:24.889159    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cca91cc-d895-4aa2-b636-8d8dd97cdbb8-kube-api-access-h79ks\" (UniqueName: \"kubernetes.io/projected/5cca91cc-d895-4aa2-b636-8d8dd97cdbb8-kube-api-access-h79ks\") pod \"5cca91cc-d895-4aa2-b636-8d8dd97cdbb8\" (UID: \"5cca91cc-d895-4aa2-b636-8d8dd97cdbb8\") "
Oct 30 01:15:24.903168 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:24.902213    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cca91cc-d895-4aa2-b636-8d8dd97cdbb8-maas-api-service-ca" pod "5cca91cc-d895-4aa2-b636-8d8dd97cdbb8" (UID: "5cca91cc-d895-4aa2-b636-8d8dd97cdbb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:24.912631 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:24.912587    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cca91cc-d895-4aa2-b636-8d8dd97cdbb8-kube-api-access-h79ks" pod "5cca91cc-d895-4aa2-b636-8d8dd97cdbb8" (UID: "5cca91cc-d895-4aa2-b636-8d8dd97cdbb8"). InnerVolumeSpecName "kube-api-access-h79ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:24.937194 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:24.937168    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl"
Oct 30 01:15:25.006142 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.006112    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h79ks\" (UniqueName: \"kubernetes.io/projected/5cca91cc-d895-4aa2-b636-8d8dd97cdbb8-kube-api-access-h79ks\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:25.006334 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.006306    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cca91cc-d895-4aa2-b636-8d8dd97cdbb8-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:25.027907 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.027883    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw"
Oct 30 01:15:25.108114 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.108022    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc4a7cce-cbf8-43b2-99cd-e7a82ab54824-kube-api-access-sz7xz\" (UniqueName: \"kubernetes.io/projected/dc4a7cce-cbf8-43b2-99cd-e7a82ab54824-kube-api-access-sz7xz\") pod \"dc4a7cce-cbf8-43b2-99cd-e7a82ab54824\" (UID: \"dc4a7cce-cbf8-43b2-99cd-e7a82ab54824\") "
Oct 30 01:15:25.108857 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.108230    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc4a7cce-cbf8-43b2-99cd-e7a82ab54824-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4a7cce-cbf8-43b2-99cd-e7a82ab54824-maas-api-service-ca\") pod \"dc4a7cce-cbf8-43b2-99cd-e7a82ab54824\" (UID: \"dc4a7cce-cbf8-43b2-99cd-e7a82ab54824\") "
Oct 30 01:15:25.108857 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.108625    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4a7cce-cbf8-43b2-99cd-e7a82ab54824-maas-api-service-ca" pod "dc4a7cce-cbf8-43b2-99cd-e7a82ab54824" (UID: "dc4a7cce-cbf8-43b2-99cd-e7a82ab54824"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:25.109191 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.109166    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4a7cce-cbf8-43b2-99cd-e7a82ab54824-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:25.110515 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.110487    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4a7cce-cbf8-43b2-99cd-e7a82ab54824-kube-api-access-sz7xz" pod "dc4a7cce-cbf8-43b2-99cd-e7a82ab54824" (UID: "dc4a7cce-cbf8-43b2-99cd-e7a82ab54824"). InnerVolumeSpecName "kube-api-access-sz7xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:25.210112 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.210082    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c09d93-afed-4328-b7a4-23775614c03c-kube-api-access-bwmh9\" (UniqueName: \"kubernetes.io/projected/07c09d93-afed-4328-b7a4-23775614c03c-kube-api-access-bwmh9\") pod \"07c09d93-afed-4328-b7a4-23775614c03c\" (UID: \"07c09d93-afed-4328-b7a4-23775614c03c\") "
Oct 30 01:15:25.210316 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.210144    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07c09d93-afed-4328-b7a4-23775614c03c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c09d93-afed-4328-b7a4-23775614c03c-maas-api-service-ca\") pod \"07c09d93-afed-4328-b7a4-23775614c03c\" (UID: \"07c09d93-afed-4328-b7a4-23775614c03c\") "
Oct 30 01:15:25.210561 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.210533    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c09d93-afed-4328-b7a4-23775614c03c-maas-api-service-ca" pod "07c09d93-afed-4328-b7a4-23775614c03c" (UID: "07c09d93-afed-4328-b7a4-23775614c03c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:25.211204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.211179    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz7xz\" (UniqueName: \"kubernetes.io/projected/dc4a7cce-cbf8-43b2-99cd-e7a82ab54824-kube-api-access-sz7xz\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:25.211204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.211202    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c09d93-afed-4328-b7a4-23775614c03c-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:25.212413 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.212392    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c09d93-afed-4328-b7a4-23775614c03c-kube-api-access-bwmh9" pod "07c09d93-afed-4328-b7a4-23775614c03c" (UID: "07c09d93-afed-4328-b7a4-23775614c03c"). InnerVolumeSpecName "kube-api-access-bwmh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:25.313235 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.313201    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwmh9\" (UniqueName: \"kubernetes.io/projected/07c09d93-afed-4328-b7a4-23775614c03c-kube-api-access-bwmh9\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:15:25.504498 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.504384    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl"
Oct 30 01:15:25.504498 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.504394    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl" event={"ID":"dc4a7cce-cbf8-43b2-99cd-e7a82ab54824","Type":"ContainerDied","Data":"cc93a47aa9f7ef791b05b8763304b338323b7317c301189707f0388571d0ce2d"}
Oct 30 01:15:25.504832 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.504763    6099 scope.go:122] "RemoveContainer" containerID="fab61c4b527c94dcbf29e282a4c274c718bd10bc394645a5d4fda41198534604"
Oct 30 01:15:25.511355 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.511335    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d"
Oct 30 01:15:25.511482 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.511349    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d" event={"ID":"5cca91cc-d895-4aa2-b636-8d8dd97cdbb8","Type":"ContainerDied","Data":"15003d4d1fc11cd9e24bc9fcd52d6143fb342b944c1248aab4e5802b9c94ff99"}
Oct 30 01:15:25.515406 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.515379    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw" event={"ID":"07c09d93-afed-4328-b7a4-23775614c03c","Type":"ContainerDied","Data":"3c832147d0de36ac5098dc30e9fbac585fb2cdb26bcc690e5282541e302e7903"}
Oct 30 01:15:25.515511 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.515471    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw"
Oct 30 01:15:25.525580 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.525326    6099 scope.go:122] "RemoveContainer" containerID="b88b2180e91e4f7cfca8f3b84a03ec2163f13b4ff8f16dd651806e37d94e442a"
Oct 30 01:15:25.537229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.537210    6099 scope.go:122] "RemoveContainer" containerID="ca4453415a4e2612f5de98f8069c59faa657eda9dbb244a48073d0b9970d5cf0"
Oct 30 01:15:25.595033 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.594990    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl"]
Oct 30 01:15:25.599843 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.599815    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-969b92-29888715-jlndl"]
Oct 30 01:15:25.631115 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.631086    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw"]
Oct 30 01:15:25.635697 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.635672    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-33e759-29888715-hzmzw"]
Oct 30 01:15:25.692986 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.692940    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d"]
Oct 30 01:15:25.698269 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:25.698240    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6add35-29888715-tkn5d"]
Oct 30 01:15:26.286732 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:26.266410    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07c09d93-afed-4328-b7a4-23775614c03c" path="/var/lib/kubelet/pods/07c09d93-afed-4328-b7a4-23775614c03c/volumes"
Oct 30 01:15:26.290329 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:26.290297    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf83038-adae-454f-81e4-5ae0e19c660b" path="/var/lib/kubelet/pods/2cf83038-adae-454f-81e4-5ae0e19c660b/volumes"
Oct 30 01:15:26.291079 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:26.291056    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cca91cc-d895-4aa2-b636-8d8dd97cdbb8" path="/var/lib/kubelet/pods/5cca91cc-d895-4aa2-b636-8d8dd97cdbb8/volumes"
Oct 30 01:15:26.291695 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:26.291676    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc4a7cce-cbf8-43b2-99cd-e7a82ab54824" path="/var/lib/kubelet/pods/dc4a7cce-cbf8-43b2-99cd-e7a82ab54824/volumes"
Oct 30 01:15:28.954931 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:28.954837    6099 patch_prober.go:29] interesting pod/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:15:28.955339 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:28.954927    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m" podUID="d8655eb2-c236-49f0-90e8-b7b4ee6e51be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:15:39.652229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.652191    6099 generic.go:356] "Generic (PLEG): container finished" podID="d8655eb2-c236-49f0-90e8-b7b4ee6e51be" containerID="b75dd08ca618e2049d074d706ebdf6720db7fd88718488d2f74f404fdb7c5c66" exitCode=0
Oct 30 01:15:39.652800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.652260    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m" event={"ID":"d8655eb2-c236-49f0-90e8-b7b4ee6e51be","Type":"ContainerDied","Data":"b75dd08ca618e2049d074d706ebdf6720db7fd88718488d2f74f404fdb7c5c66"}
Oct 30 01:15:39.652910 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.652839    6099 scope.go:122] "RemoveContainer" containerID="b75dd08ca618e2049d074d706ebdf6720db7fd88718488d2f74f404fdb7c5c66"
Oct 30 01:15:39.654987 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.654958    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:15:39.655067 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.655042    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:15:39.655107 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.655073    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:15:39.890791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.890753    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"]
Oct 30 01:15:39.892081 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892054    6099 cpu_manager.go:414] "containerMap: removing container" podUID="7c9e6dd5-8118-494e-8cfd-bab30c938ddc" containerName="cleanup"
Oct 30 01:15:39.892271 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892248    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9e6dd5-8118-494e-8cfd-bab30c938ddc" containerName="cleanup"
Oct 30 01:15:39.892350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892280    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5cca91cc-d895-4aa2-b636-8d8dd97cdbb8" containerName="cleanup"
Oct 30 01:15:39.892350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892286    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cca91cc-d895-4aa2-b636-8d8dd97cdbb8" containerName="cleanup"
Oct 30 01:15:39.892350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892297    6099 cpu_manager.go:414] "containerMap: removing container" podUID="75974319-3673-4bdc-bb2c-e9be475a5723" containerName="cleanup"
Oct 30 01:15:39.892350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892302    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75974319-3673-4bdc-bb2c-e9be475a5723" containerName="cleanup"
Oct 30 01:15:39.892350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892308    6099 cpu_manager.go:414] "containerMap: removing container" podUID="2cf83038-adae-454f-81e4-5ae0e19c660b" containerName="cleanup"
Oct 30 01:15:39.892350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892315    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf83038-adae-454f-81e4-5ae0e19c660b" containerName="cleanup"
Oct 30 01:15:39.892350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892348    6099 cpu_manager.go:414] "containerMap: removing container" podUID="b64df5bf-b9b0-47f9-bbd5-7c2839b42a53" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892356    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64df5bf-b9b0-47f9-bbd5-7c2839b42a53" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892376    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5cca91cc-d895-4aa2-b636-8d8dd97cdbb8" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892385    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cca91cc-d895-4aa2-b636-8d8dd97cdbb8" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892410    6099 cpu_manager.go:414] "containerMap: removing container" podUID="40f9d853-515b-48e7-b72b-5fd68d43ac4f" containerName="payload-processing"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892417    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f9d853-515b-48e7-b72b-5fd68d43ac4f" containerName="payload-processing"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892432    6099 cpu_manager.go:414] "containerMap: removing container" podUID="9ffde68d-ddf9-439e-9cff-b0aab29c397e" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892439    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ffde68d-ddf9-439e-9cff-b0aab29c397e" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892454    6099 cpu_manager.go:414] "containerMap: removing container" podUID="9e96d7ed-1b43-45d9-9d84-64b9aaf7e694" containerName="main"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892461    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e96d7ed-1b43-45d9-9d84-64b9aaf7e694" containerName="main"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892474    6099 cpu_manager.go:414] "containerMap: removing container" podUID="7c9e6dd5-8118-494e-8cfd-bab30c938ddc" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892480    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9e6dd5-8118-494e-8cfd-bab30c938ddc" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892491    6099 cpu_manager.go:414] "containerMap: removing container" podUID="75974319-3673-4bdc-bb2c-e9be475a5723" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892502    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75974319-3673-4bdc-bb2c-e9be475a5723" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892513    6099 cpu_manager.go:414] "containerMap: removing container" podUID="9ffde68d-ddf9-439e-9cff-b0aab29c397e" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892519    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ffde68d-ddf9-439e-9cff-b0aab29c397e" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892555    6099 cpu_manager.go:414] "containerMap: removing container" podUID="dc4a7cce-cbf8-43b2-99cd-e7a82ab54824" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892582    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc4a7cce-cbf8-43b2-99cd-e7a82ab54824" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892605    6099 cpu_manager.go:414] "containerMap: removing container" podUID="07c09d93-afed-4328-b7a4-23775614c03c" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892612    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c09d93-afed-4328-b7a4-23775614c03c" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892630    6099 cpu_manager.go:414] "containerMap: removing container" podUID="dc4a7cce-cbf8-43b2-99cd-e7a82ab54824" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892637    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc4a7cce-cbf8-43b2-99cd-e7a82ab54824" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892657    6099 cpu_manager.go:414] "containerMap: removing container" podUID="dc4a7cce-cbf8-43b2-99cd-e7a82ab54824" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892664    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc4a7cce-cbf8-43b2-99cd-e7a82ab54824" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892753    6099 cpu_manager.go:414] "containerMap: removing container" podUID="0b9a2818-c031-4bae-b1a2-d8793f88c00f" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892760    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9a2818-c031-4bae-b1a2-d8793f88c00f" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892770    6099 cpu_manager.go:414] "containerMap: removing container" podUID="80a38635-cb84-404c-9502-de7b43239b8e" containerName="istio-proxy"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892777    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a38635-cb84-404c-9502-de7b43239b8e" containerName="istio-proxy"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892787    6099 cpu_manager.go:414] "containerMap: removing container" podUID="0b9a2818-c031-4bae-b1a2-d8793f88c00f" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892793    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9a2818-c031-4bae-b1a2-d8793f88c00f" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892810    6099 cpu_manager.go:414] "containerMap: removing container" podUID="80a38635-cb84-404c-9502-de7b43239b8e" containerName="istio-proxy"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892816    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a38635-cb84-404c-9502-de7b43239b8e" containerName="istio-proxy"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892831    6099 cpu_manager.go:414] "containerMap: removing container" podUID="07c09d93-afed-4328-b7a4-23775614c03c" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892837    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c09d93-afed-4328-b7a4-23775614c03c" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892849    6099 cpu_manager.go:414] "containerMap: removing container" podUID="2cf83038-adae-454f-81e4-5ae0e19c660b" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892856    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf83038-adae-454f-81e4-5ae0e19c660b" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892870    6099 cpu_manager.go:414] "containerMap: removing container" podUID="0b9a2818-c031-4bae-b1a2-d8793f88c00f" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892877    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9a2818-c031-4bae-b1a2-d8793f88c00f" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892889    6099 cpu_manager.go:414] "containerMap: removing container" podUID="b64df5bf-b9b0-47f9-bbd5-7c2839b42a53" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.892897    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64df5bf-b9b0-47f9-bbd5-7c2839b42a53" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893136    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f9d853-515b-48e7-b72b-5fd68d43ac4f" containerName="payload-processing"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893155    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ffde68d-ddf9-439e-9cff-b0aab29c397e" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893165    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e96d7ed-1b43-45d9-9d84-64b9aaf7e694" containerName="main"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893174    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9e6dd5-8118-494e-8cfd-bab30c938ddc" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893181    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="75974319-3673-4bdc-bb2c-e9be475a5723" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893188    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ffde68d-ddf9-439e-9cff-b0aab29c397e" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893199    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc4a7cce-cbf8-43b2-99cd-e7a82ab54824" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893210    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c09d93-afed-4328-b7a4-23775614c03c" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893218    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc4a7cce-cbf8-43b2-99cd-e7a82ab54824" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893227    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc4a7cce-cbf8-43b2-99cd-e7a82ab54824" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893245    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cca91cc-d895-4aa2-b636-8d8dd97cdbb8" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893257    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9a2818-c031-4bae-b1a2-d8793f88c00f" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893265    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a38635-cb84-404c-9502-de7b43239b8e" containerName="istio-proxy"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893275    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9a2818-c031-4bae-b1a2-d8793f88c00f" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893285    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64df5bf-b9b0-47f9-bbd5-7c2839b42a53" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893296    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a38635-cb84-404c-9502-de7b43239b8e" containerName="istio-proxy"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893308    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c09d93-afed-4328-b7a4-23775614c03c" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893317    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf83038-adae-454f-81e4-5ae0e19c660b" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893324    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf83038-adae-454f-81e4-5ae0e19c660b" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893333    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64df5bf-b9b0-47f9-bbd5-7c2839b42a53" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893342    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="75974319-3673-4bdc-bb2c-e9be475a5723" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893351    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9e6dd5-8118-494e-8cfd-bab30c938ddc" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893360    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cca91cc-d895-4aa2-b636-8d8dd97cdbb8" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893369    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="75974319-3673-4bdc-bb2c-e9be475a5723" containerName="cleanup"
Oct 30 01:15:39.894072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.893377    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64df5bf-b9b0-47f9-bbd5-7c2839b42a53" containerName="cleanup"
Oct 30 01:15:39.897545 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.897521    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:39.917596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.917569    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"]
Oct 30 01:15:39.996583 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.996530    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-workload-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:39.996800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.996633    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1533e0-8ff9-4108-8688-714e05957447-istiod-ca-cert\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:39.996800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.996748    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-credential-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:39.996800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.996781    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-workload-certs\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:39.996999 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.996835    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-istio-data\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:39.996999 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.996883    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1533e0-8ff9-4108-8688-714e05957447-istio-token\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:39.997113 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.997059    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-istio-envoy\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:39.997113 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.997098    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpprh\" (UniqueName: \"kubernetes.io/projected/7c1533e0-8ff9-4108-8688-714e05957447-kube-api-access-dpprh\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:39.997226 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:39.997203    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1533e0-8ff9-4108-8688-714e05957447-istio-podinfo\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.098532 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.098491    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-istio-envoy\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.098762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.098542    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpprh\" (UniqueName: \"kubernetes.io/projected/7c1533e0-8ff9-4108-8688-714e05957447-kube-api-access-dpprh\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.098762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.098682    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1533e0-8ff9-4108-8688-714e05957447-istio-podinfo\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.098909 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.098854    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-workload-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.098974 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.098923    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1533e0-8ff9-4108-8688-714e05957447-istiod-ca-cert\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.099034 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.099018    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-credential-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.099095 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.099045    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-workload-certs\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.099162 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.099091    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-istio-data\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.099162 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.099137    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1533e0-8ff9-4108-8688-714e05957447-istio-token\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.099322 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.099296    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-workload-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.099554 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.099523    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-credential-socket\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.099700 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.099584    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-istio-data\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.099700 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.099673    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-workload-certs\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.101949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.101917    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-istio-envoy\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.102216 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.102192    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1533e0-8ff9-4108-8688-714e05957447-istiod-ca-cert\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.102412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.102388    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1533e0-8ff9-4108-8688-714e05957447-istio-podinfo\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.122299 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.122270    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpprh\" (UniqueName: \"kubernetes.io/projected/7c1533e0-8ff9-4108-8688-714e05957447-kube-api-access-dpprh\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.122299 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.122284    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1533e0-8ff9-4108-8688-714e05957447-istio-token\") pod \"e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") " pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.218059 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.217964    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:40.589877 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.589832    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"]
Oct 30 01:15:40.611697 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:15:40.611629    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1533e0_8ff9_4108_8688_714e05957447.slice/crio-afd7cad66432ecbdc5614084a409b36c82ec486c4c978ef474545f29f1368323 WatchSource:0}: Error finding container afd7cad66432ecbdc5614084a409b36c82ec486c4c978ef474545f29f1368323: Status 404 returned error can't find the container with id afd7cad66432ecbdc5614084a409b36c82ec486c4c978ef474545f29f1368323
Oct 30 01:15:40.615854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.615814    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:15:40.615986 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.615905    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:15:40.615986 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.615953    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:15:40.669642 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.668355    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m" event={"ID":"d8655eb2-c236-49f0-90e8-b7b4ee6e51be","Type":"ContainerStarted","Data":"b503fc5080acd83abef3b4b7395e0ce885bde57e2d7accb4659ee5b7ab24faaf"}
Oct 30 01:15:40.671985 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:40.671917    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2" event={"ID":"7c1533e0-8ff9-4108-8688-714e05957447","Type":"ContainerStarted","Data":"afd7cad66432ecbdc5614084a409b36c82ec486c4c978ef474545f29f1368323"}
Oct 30 01:15:41.505750 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:41.502985    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:15:41.516729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:41.509608    6099 patch_prober.go:29] interesting pod/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 30 01:15:41.516729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:41.509678    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m" podUID="d8655eb2-c236-49f0-90e8-b7b4ee6e51be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 30 01:15:41.732428 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:41.732378    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2" event={"ID":"7c1533e0-8ff9-4108-8688-714e05957447","Type":"ContainerStarted","Data":"743d37801d77c41b9983afcc6580305757b60c256a94a7d43968574a9bafe0ae"}
Oct 30 01:15:41.822780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:41.809914    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2" podStartSLOduration=2.80989045 podStartE2EDuration="2.80989045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:41.7811955 +0000 UTC m=+2435.638103279" watchObservedRunningTime="2026-10-30 01:15:41.80989045 +0000 UTC m=+2435.666798209"
Oct 30 01:15:42.231537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:42.231431    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:42.233754 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:42.232854    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body=
Oct 30 01:15:42.233754 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:42.232911    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2" podUID="7c1533e0-8ff9-4108-8688-714e05957447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused"
Oct 30 01:15:42.515787 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:42.514877    6099 patch_prober.go:29] interesting pod/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 30 01:15:42.515787 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:42.514947    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m" podUID="d8655eb2-c236-49f0-90e8-b7b4ee6e51be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 30 01:15:42.914730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:42.914064    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:15:43.220512 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.220109    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body=
Oct 30 01:15:43.220512 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.220198    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2" podUID="7c1533e0-8ff9-4108-8688-714e05957447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused"
Oct 30 01:15:43.503146 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.503031    6099 patch_prober.go:29] interesting pod/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 30 01:15:43.503896 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.503851    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m" podUID="d8655eb2-c236-49f0-90e8-b7b4ee6e51be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 30 01:15:43.939377 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.939343    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"]
Oct 30 01:15:43.940199 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940181    6099 cpu_manager.go:414] "containerMap: removing container" podUID="7c9e6dd5-8118-494e-8cfd-bab30c938ddc" containerName="cleanup"
Oct 30 01:15:43.940199 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940199    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9e6dd5-8118-494e-8cfd-bab30c938ddc" containerName="cleanup"
Oct 30 01:15:43.940346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940213    6099 cpu_manager.go:414] "containerMap: removing container" podUID="75974319-3673-4bdc-bb2c-e9be475a5723" containerName="cleanup"
Oct 30 01:15:43.940346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940218    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75974319-3673-4bdc-bb2c-e9be475a5723" containerName="cleanup"
Oct 30 01:15:43.940346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940226    6099 cpu_manager.go:414] "containerMap: removing container" podUID="9ffde68d-ddf9-439e-9cff-b0aab29c397e" containerName="cleanup"
Oct 30 01:15:43.940346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940231    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ffde68d-ddf9-439e-9cff-b0aab29c397e" containerName="cleanup"
Oct 30 01:15:43.940346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940274    6099 cpu_manager.go:414] "containerMap: removing container" podUID="07c09d93-afed-4328-b7a4-23775614c03c" containerName="cleanup"
Oct 30 01:15:43.940346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940279    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c09d93-afed-4328-b7a4-23775614c03c" containerName="cleanup"
Oct 30 01:15:43.940346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940295    6099 cpu_manager.go:414] "containerMap: removing container" podUID="2cf83038-adae-454f-81e4-5ae0e19c660b" containerName="cleanup"
Oct 30 01:15:43.940346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940299    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf83038-adae-454f-81e4-5ae0e19c660b" containerName="cleanup"
Oct 30 01:15:43.940346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940307    6099 cpu_manager.go:414] "containerMap: removing container" podUID="b64df5bf-b9b0-47f9-bbd5-7c2839b42a53" containerName="cleanup"
Oct 30 01:15:43.940346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940312    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64df5bf-b9b0-47f9-bbd5-7c2839b42a53" containerName="cleanup"
Oct 30 01:15:43.940346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940320    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5cca91cc-d895-4aa2-b636-8d8dd97cdbb8" containerName="cleanup"
Oct 30 01:15:43.940346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940324    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cca91cc-d895-4aa2-b636-8d8dd97cdbb8" containerName="cleanup"
Oct 30 01:15:43.940780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940445    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9a2818-c031-4bae-b1a2-d8793f88c00f" containerName="cleanup"
Oct 30 01:15:43.940780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940457    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf83038-adae-454f-81e4-5ae0e19c660b" containerName="cleanup"
Oct 30 01:15:43.940780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940465    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9e6dd5-8118-494e-8cfd-bab30c938ddc" containerName="cleanup"
Oct 30 01:15:43.940780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940472    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cca91cc-d895-4aa2-b636-8d8dd97cdbb8" containerName="cleanup"
Oct 30 01:15:43.940780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940481    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ffde68d-ddf9-439e-9cff-b0aab29c397e" containerName="cleanup"
Oct 30 01:15:43.940780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.940487    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c09d93-afed-4328-b7a4-23775614c03c" containerName="cleanup"
Oct 30 01:15:43.943683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.943664    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:43.990735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.964963    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f42de42-9772-41e0-9cdb-8a9b212106ee-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:43.990735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.965027    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:43.990735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.965106    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:43.990735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.965212    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:43.990735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.965261    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:43.990735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.965310    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:43.990735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.965345    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnbjk\" (UniqueName: \"kubernetes.io/projected/4f42de42-9772-41e0-9cdb-8a9b212106ee-kube-api-access-xnbjk\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:43.990735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.965397    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:43.990735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:43.965444    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.035769 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.030404    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"]
Oct 30 01:15:44.119806 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.110440    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.120019 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.119923    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.120019 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.120005    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.120185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.120071    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.120185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.120109    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnbjk\" (UniqueName: \"kubernetes.io/projected/4f42de42-9772-41e0-9cdb-8a9b212106ee-kube-api-access-xnbjk\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.120185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.120172    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.120382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.120227    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.120519 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.120492    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f42de42-9772-41e0-9cdb-8a9b212106ee-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.120587 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.120560    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.121197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.121168    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-credential-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.121330 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.111189    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-workload-certs\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.121330 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.121268    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-workload-socket\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.121518 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.121494    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f42de42-9772-41e0-9cdb-8a9b212106ee-istiod-ca-cert\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.121872 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.121849    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-data\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.123767 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.123746    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-envoy\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.124360 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.124336    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-podinfo\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.144691 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.144660    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-token\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.144863 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.144842    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnbjk\" (UniqueName: \"kubernetes.io/projected/4f42de42-9772-41e0-9cdb-8a9b212106ee-kube-api-access-xnbjk\") pod \"e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") " pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.220227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.219287    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body=
Oct 30 01:15:44.220227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.219365    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2" podUID="7c1533e0-8ff9-4108-8688-714e05957447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused"
Oct 30 01:15:44.260198 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.260166    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:44.564354 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.559806    6099 patch_prober.go:29] interesting pod/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 30 01:15:44.564354 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.559861    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m" podUID="d8655eb2-c236-49f0-90e8-b7b4ee6e51be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 30 01:15:44.722963 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.722931    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"]
Oct 30 01:15:44.724539 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:15:44.724490    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f42de42_9772_41e0_9cdb_8a9b212106ee.slice/crio-7aff112a334002f3aea1226921fbf581be492088b2c9f4681989e7df408209d4 WatchSource:0}: Error finding container 7aff112a334002f3aea1226921fbf581be492088b2c9f4681989e7df408209d4: Status 404 returned error can't find the container with id 7aff112a334002f3aea1226921fbf581be492088b2c9f4681989e7df408209d4
Oct 30 01:15:44.733754 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.733694    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:15:44.733888 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.733803    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:15:44.733888 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.733846    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:15:44.763094 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:44.763063    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg" event={"ID":"4f42de42-9772-41e0-9cdb-8a9b212106ee","Type":"ContainerStarted","Data":"7aff112a334002f3aea1226921fbf581be492088b2c9f4681989e7df408209d4"}
Oct 30 01:15:45.224788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:45.219800    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body=
Oct 30 01:15:45.224788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:45.219862    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2" podUID="7c1533e0-8ff9-4108-8688-714e05957447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused"
Oct 30 01:15:45.509789 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:45.509192    6099 patch_prober.go:29] interesting pod/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 30 01:15:45.509789 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:45.509253    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m" podUID="d8655eb2-c236-49f0-90e8-b7b4ee6e51be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 30 01:15:45.816729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:45.812964    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg" event={"ID":"4f42de42-9772-41e0-9cdb-8a9b212106ee","Type":"ContainerStarted","Data":"73e0372ba6604553dadd0e370509ed9ee7cc6dc08a04333fac52e458ca36a7d0"}
Oct 30 01:15:45.906729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:45.906458    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg" podStartSLOduration=2.906436757 podStartE2EDuration="2.906436757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:45.884230255 +0000 UTC m=+2439.741138035" watchObservedRunningTime="2026-10-30 01:15:45.906436757 +0000 UTC m=+2439.763344512"
Oct 30 01:15:46.227786 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:46.221687    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body=
Oct 30 01:15:46.227786 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:46.221767    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2" podUID="7c1533e0-8ff9-4108-8688-714e05957447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused"
Oct 30 01:15:46.309371 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:46.279185    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:46.309371 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:46.279320    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" start-of-body=
Oct 30 01:15:46.309371 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:46.279361    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg" podUID="4f42de42-9772-41e0-9cdb-8a9b212106ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused"
Oct 30 01:15:46.523017 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:46.522681    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:15:46.525030 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:46.525008    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:15:46.658453 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:15:46.658417    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:15:47.218401 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:47.218345    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body=
Oct 30 01:15:47.218679 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:47.218409    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2" podUID="7c1533e0-8ff9-4108-8688-714e05957447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused"
Oct 30 01:15:47.260639 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:47.260604    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" start-of-body=
Oct 30 01:15:47.261209 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:47.260662    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg" podUID="4f42de42-9772-41e0-9cdb-8a9b212106ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused"
Oct 30 01:15:48.224572 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:48.224491    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:48.225007 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:48.224983    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:48.225820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:48.225798    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:15:48.260468 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:48.260445    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" start-of-body=
Oct 30 01:15:48.260560 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:48.260498    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg" podUID="4f42de42-9772-41e0-9cdb-8a9b212106ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused"
Oct 30 01:15:49.260913 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:49.260886    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" start-of-body=
Oct 30 01:15:49.261319 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:49.260933    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg" podUID="4f42de42-9772-41e0-9cdb-8a9b212106ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused"
Oct 30 01:15:50.260823 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:50.260793    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" start-of-body=
Oct 30 01:15:50.260996 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:50.260846    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg" podUID="4f42de42-9772-41e0-9cdb-8a9b212106ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused"
Oct 30 01:15:51.261406 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.261369    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" start-of-body=
Oct 30 01:15:51.261944 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.261438    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg" podUID="4f42de42-9772-41e0-9cdb-8a9b212106ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused"
Oct 30 01:15:51.399447 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.399415    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"]
Oct 30 01:15:51.414379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.414334    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"
Oct 30 01:15:51.436725 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.436668    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"]
Oct 30 01:15:51.441033 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.441007    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d1ae36-serving-cert\""
Oct 30 01:15:51.574580 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.574547    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wct6f\" (UniqueName: \"kubernetes.io/projected/75cc8389-4904-44b2-8c92-cf3debf351f4-kube-api-access-wct6f\") pod \"maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6\" (UID: \"75cc8389-4904-44b2-8c92-cf3debf351f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"
Oct 30 01:15:51.574794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.574699    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cc8389-4904-44b2-8c92-cf3debf351f4-metrics-certs\") pod \"maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6\" (UID: \"75cc8389-4904-44b2-8c92-cf3debf351f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"
Oct 30 01:15:51.574794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.574766    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75cc8389-4904-44b2-8c92-cf3debf351f4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6\" (UID: \"75cc8389-4904-44b2-8c92-cf3debf351f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"
Oct 30 01:15:51.574920 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.574908    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cc8389-4904-44b2-8c92-cf3debf351f4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6\" (UID: \"75cc8389-4904-44b2-8c92-cf3debf351f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"
Oct 30 01:15:51.676455 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.676414    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wct6f\" (UniqueName: \"kubernetes.io/projected/75cc8389-4904-44b2-8c92-cf3debf351f4-kube-api-access-wct6f\") pod \"maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6\" (UID: \"75cc8389-4904-44b2-8c92-cf3debf351f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"
Oct 30 01:15:51.676669 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.676540    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cc8389-4904-44b2-8c92-cf3debf351f4-metrics-certs\") pod \"maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6\" (UID: \"75cc8389-4904-44b2-8c92-cf3debf351f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"
Oct 30 01:15:51.676669 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.676562    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75cc8389-4904-44b2-8c92-cf3debf351f4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6\" (UID: \"75cc8389-4904-44b2-8c92-cf3debf351f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"
Oct 30 01:15:51.676837 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.676695    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cc8389-4904-44b2-8c92-cf3debf351f4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6\" (UID: \"75cc8389-4904-44b2-8c92-cf3debf351f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"
Oct 30 01:15:51.677503 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.677480    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cc8389-4904-44b2-8c92-cf3debf351f4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6\" (UID: \"75cc8389-4904-44b2-8c92-cf3debf351f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"
Oct 30 01:15:51.679663 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.679640    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cc8389-4904-44b2-8c92-cf3debf351f4-metrics-certs\") pod \"maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6\" (UID: \"75cc8389-4904-44b2-8c92-cf3debf351f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"
Oct 30 01:15:51.679863 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.679839    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75cc8389-4904-44b2-8c92-cf3debf351f4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6\" (UID: \"75cc8389-4904-44b2-8c92-cf3debf351f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"
Oct 30 01:15:51.699383 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.699361    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wct6f\" (UniqueName: \"kubernetes.io/projected/75cc8389-4904-44b2-8c92-cf3debf351f4-kube-api-access-wct6f\") pod \"maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6\" (UID: \"75cc8389-4904-44b2-8c92-cf3debf351f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"
Oct 30 01:15:51.752822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:51.752113    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"
Oct 30 01:15:52.266417 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:52.266385    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:52.266948 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:52.266926    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:52.268931 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:52.268907    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:15:52.361345 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:52.361253    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"]
Oct 30 01:15:52.371491 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:15:52.371455    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75cc8389_4904_44b2_8c92_cf3debf351f4.slice/crio-3ea62a5114b0c3da97e311f0f562d562682a676dccc7338ef6a936ef67ef05f7 WatchSource:0}: Error finding container 3ea62a5114b0c3da97e311f0f562d562682a676dccc7338ef6a936ef67ef05f7: Status 404 returned error can't find the container with id 3ea62a5114b0c3da97e311f0f562d562682a676dccc7338ef6a936ef67ef05f7
Oct 30 01:15:52.373513 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:52.373493    6099 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:15:52.983430 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:52.983393    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6" event={"ID":"75cc8389-4904-44b2-8c92-cf3debf351f4","Type":"ContainerStarted","Data":"6359d9b2d450528e76ccf389945b98abdca2178714bfa1d908e81beecb8f2027"}
Oct 30 01:15:52.983635 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:52.983450    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6" event={"ID":"75cc8389-4904-44b2-8c92-cf3debf351f4","Type":"ContainerStarted","Data":"3ea62a5114b0c3da97e311f0f562d562682a676dccc7338ef6a936ef67ef05f7"}
Oct 30 01:15:52.983635 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:52.983484    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"
Oct 30 01:15:53.028534 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:53.028430    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6" podStartSLOduration=1.648236061 podStartE2EDuration="2.028415674s" totalImagesPullingTime="380.179613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:53.025699832 +0000 UTC m=+2446.882607587" watchObservedRunningTime="2026-10-30 01:15:53.028415674 +0000 UTC m=+2446.885323428"
Oct 30 01:15:58.999789 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:15:58.999759    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"
Oct 30 01:16:02.400912 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:02.400878    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"]
Oct 30 01:16:02.401374 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:02.401094    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q" podUID="1317f970-0225-46f8-bf6f-a5e6be570cab" containerName="istio-proxy" containerID="cri-o://20caef8c899bfcfb47124e154244780b0d70ac48ed00fa805eaaa18e547d8646" gracePeriod=30
Oct 30 01:16:07.699135 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.699105    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:16:07.860086 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.860054    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1317f970-0225-46f8-bf6f-a5e6be570cab-kube-api-access-x68v7\" (UniqueName: \"kubernetes.io/projected/1317f970-0225-46f8-bf6f-a5e6be570cab-kube-api-access-x68v7\") pod \"1317f970-0225-46f8-bf6f-a5e6be570cab\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") "
Oct 30 01:16:07.860086 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.860096    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1317f970-0225-46f8-bf6f-a5e6be570cab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1317f970-0225-46f8-bf6f-a5e6be570cab-istiod-ca-cert\") pod \"1317f970-0225-46f8-bf6f-a5e6be570cab\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") "
Oct 30 01:16:07.860412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.860134    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-workload-socket\") pod \"1317f970-0225-46f8-bf6f-a5e6be570cab\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") "
Oct 30 01:16:07.860412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.860273    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-podinfo\") pod \"1317f970-0225-46f8-bf6f-a5e6be570cab\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") "
Oct 30 01:16:07.860412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.860396    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-envoy\") pod \"1317f970-0225-46f8-bf6f-a5e6be570cab\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") "
Oct 30 01:16:07.860584 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.860463    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-workload-socket" pod "1317f970-0225-46f8-bf6f-a5e6be570cab" (UID: "1317f970-0225-46f8-bf6f-a5e6be570cab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:07.860584 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.860524    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-credential-socket\") pod \"1317f970-0225-46f8-bf6f-a5e6be570cab\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") "
Oct 30 01:16:07.860761 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.860636    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-workload-certs\") pod \"1317f970-0225-46f8-bf6f-a5e6be570cab\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") "
Oct 30 01:16:07.860761 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.860659    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1317f970-0225-46f8-bf6f-a5e6be570cab-istiod-ca-cert" pod "1317f970-0225-46f8-bf6f-a5e6be570cab" (UID: "1317f970-0225-46f8-bf6f-a5e6be570cab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:07.860910 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.860787    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-token\" (UniqueName: \"kubernetes.io/projected/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-token\") pod \"1317f970-0225-46f8-bf6f-a5e6be570cab\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") "
Oct 30 01:16:07.860910 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.860839    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-data\") pod \"1317f970-0225-46f8-bf6f-a5e6be570cab\" (UID: \"1317f970-0225-46f8-bf6f-a5e6be570cab\") "
Oct 30 01:16:07.861031 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.861001    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-workload-certs" pod "1317f970-0225-46f8-bf6f-a5e6be570cab" (UID: "1317f970-0225-46f8-bf6f-a5e6be570cab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:07.861093 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.861074    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-credential-socket" pod "1317f970-0225-46f8-bf6f-a5e6be570cab" (UID: "1317f970-0225-46f8-bf6f-a5e6be570cab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:07.861174 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.861155    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-data" pod "1317f970-0225-46f8-bf6f-a5e6be570cab" (UID: "1317f970-0225-46f8-bf6f-a5e6be570cab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:07.862267 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.862239    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:16:07.862405 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.862271    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:16:07.862405 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.862288    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:16:07.862405 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.862302    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1317f970-0225-46f8-bf6f-a5e6be570cab-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:16:07.862405 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.862315    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:16:07.863374 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.863350    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-podinfo" pod "1317f970-0225-46f8-bf6f-a5e6be570cab" (UID: "1317f970-0225-46f8-bf6f-a5e6be570cab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:16:07.863655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.863625    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-token" pod "1317f970-0225-46f8-bf6f-a5e6be570cab" (UID: "1317f970-0225-46f8-bf6f-a5e6be570cab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:07.863844 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.863821    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1317f970-0225-46f8-bf6f-a5e6be570cab-kube-api-access-x68v7" pod "1317f970-0225-46f8-bf6f-a5e6be570cab" (UID: "1317f970-0225-46f8-bf6f-a5e6be570cab"). InnerVolumeSpecName "kube-api-access-x68v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:07.863935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.863880    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-envoy" pod "1317f970-0225-46f8-bf6f-a5e6be570cab" (UID: "1317f970-0225-46f8-bf6f-a5e6be570cab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:07.963683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.963615    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x68v7\" (UniqueName: \"kubernetes.io/projected/1317f970-0225-46f8-bf6f-a5e6be570cab-kube-api-access-x68v7\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:16:07.963683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.963645    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:16:07.963683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.963654    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:16:07.963683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:07.963662    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1317f970-0225-46f8-bf6f-a5e6be570cab-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:16:08.169124 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:08.169090    6099 generic.go:356] "Generic (PLEG): container finished" podID="1317f970-0225-46f8-bf6f-a5e6be570cab" containerID="20caef8c899bfcfb47124e154244780b0d70ac48ed00fa805eaaa18e547d8646" exitCode=0
Oct 30 01:16:08.169313 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:08.169144    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"
Oct 30 01:16:08.169313 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:08.169145    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q" event={"ID":"1317f970-0225-46f8-bf6f-a5e6be570cab","Type":"ContainerDied","Data":"20caef8c899bfcfb47124e154244780b0d70ac48ed00fa805eaaa18e547d8646"}
Oct 30 01:16:08.169313 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:08.169291    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q" event={"ID":"1317f970-0225-46f8-bf6f-a5e6be570cab","Type":"ContainerDied","Data":"c24c867875a769bc7d2b0c798a42bed7f06b90dddf695565a8aa500bc98f4e72"}
Oct 30 01:16:08.169456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:08.169330    6099 scope.go:122] "RemoveContainer" containerID="20caef8c899bfcfb47124e154244780b0d70ac48ed00fa805eaaa18e547d8646"
Oct 30 01:16:08.190084 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:08.190062    6099 scope.go:122] "RemoveContainer" containerID="20caef8c899bfcfb47124e154244780b0d70ac48ed00fa805eaaa18e547d8646"
Oct 30 01:16:08.191082 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:16:08.191025    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20caef8c899bfcfb47124e154244780b0d70ac48ed00fa805eaaa18e547d8646\": container with ID starting with 20caef8c899bfcfb47124e154244780b0d70ac48ed00fa805eaaa18e547d8646 not found: ID does not exist" containerID="20caef8c899bfcfb47124e154244780b0d70ac48ed00fa805eaaa18e547d8646"
Oct 30 01:16:08.191082 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:08.191061    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20caef8c899bfcfb47124e154244780b0d70ac48ed00fa805eaaa18e547d8646"} err="failed to get container status \"20caef8c899bfcfb47124e154244780b0d70ac48ed00fa805eaaa18e547d8646\": rpc error: code = NotFound desc = could not find container \"20caef8c899bfcfb47124e154244780b0d70ac48ed00fa805eaaa18e547d8646\": container with ID starting with 20caef8c899bfcfb47124e154244780b0d70ac48ed00fa805eaaa18e547d8646 not found: ID does not exist"
Oct 30 01:16:08.227003 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:08.226943    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"]
Oct 30 01:16:08.236359 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:08.236335    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-fc45q"]
Oct 30 01:16:08.264515 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:08.264490    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1317f970-0225-46f8-bf6f-a5e6be570cab" path="/var/lib/kubelet/pods/1317f970-0225-46f8-bf6f-a5e6be570cab/volumes"
Oct 30 01:16:10.785792 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:10.772819    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:16:13.794434 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:13.794390    6099 scope.go:122] "RemoveContainer" containerID="2e0674e7476f236d0753682c83e4fa81eb9e3378c6699a4ddcd14651935a1a9e"
Oct 30 01:16:13.842985 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:13.842959    6099 scope.go:122] "RemoveContainer" containerID="2f789618238245e770da18e731d025a3ecfed063efe5bebe117dbfb9bda72284"
Oct 30 01:16:13.957950 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:13.955923    6099 scope.go:122] "RemoveContainer" containerID="9cf327e31878fe32c1bce027595620d3304bd91b3b75182b01270fc5b6d77234"
Oct 30 01:16:14.095164 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:14.095117    6099 scope.go:122] "RemoveContainer" containerID="c703ef40d58c2f153f161fb98aa981ad26664f5ac11846954bf154e55ae1f913"
Oct 30 01:16:16.631594 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:16:16.631555    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:16:22.388247 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:22.388217    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr"]
Oct 30 01:16:22.389060 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:22.389042    6099 cpu_manager.go:414] "containerMap: removing container" podUID="1317f970-0225-46f8-bf6f-a5e6be570cab" containerName="istio-proxy"
Oct 30 01:16:22.389060 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:22.389061    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1317f970-0225-46f8-bf6f-a5e6be570cab" containerName="istio-proxy"
Oct 30 01:16:22.389204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:22.389192    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1317f970-0225-46f8-bf6f-a5e6be570cab" containerName="istio-proxy"
Oct 30 01:16:22.394936 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:22.394912    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr"]
Oct 30 01:16:22.395076 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:22.395024    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr"
Oct 30 01:16:22.414954 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:22.412903    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1a42e5\""
Oct 30 01:16:22.423980 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:22.423955    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1a42e5-dockercfg-khx2x\""
Oct 30 01:16:22.479911 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:22.479867    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5792\" (UniqueName: \"kubernetes.io/projected/a627558d-fddd-4135-be1e-b24b424d0479-kube-api-access-x5792\") pod \"payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr\" (UID: \"a627558d-fddd-4135-be1e-b24b424d0479\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr"
Oct 30 01:16:22.480122 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:22.479937    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a627558d-fddd-4135-be1e-b24b424d0479-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr\" (UID: \"a627558d-fddd-4135-be1e-b24b424d0479\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr"
Oct 30 01:16:22.589235 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:22.589196    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5792\" (UniqueName: \"kubernetes.io/projected/a627558d-fddd-4135-be1e-b24b424d0479-kube-api-access-x5792\") pod \"payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr\" (UID: \"a627558d-fddd-4135-be1e-b24b424d0479\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr"
Oct 30 01:16:22.589442 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:22.589273    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a627558d-fddd-4135-be1e-b24b424d0479-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr\" (UID: \"a627558d-fddd-4135-be1e-b24b424d0479\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr"
Oct 30 01:16:22.590322 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:22.590297    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a627558d-fddd-4135-be1e-b24b424d0479-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr\" (UID: \"a627558d-fddd-4135-be1e-b24b424d0479\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr"
Oct 30 01:16:22.612875 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:22.612843    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5792\" (UniqueName: \"kubernetes.io/projected/a627558d-fddd-4135-be1e-b24b424d0479-kube-api-access-x5792\") pod \"payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr\" (UID: \"a627558d-fddd-4135-be1e-b24b424d0479\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr"
Oct 30 01:16:22.723012 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:22.722914    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr"
Oct 30 01:16:23.048902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:23.048868    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr"]
Oct 30 01:16:23.049597 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:16:23.049568    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda627558d_fddd_4135_be1e_b24b424d0479.slice/crio-2f2f6d93e8aff45b1e689ebf9cc8210f0a271beed17e334a749bb2522d85d618 WatchSource:0}: Error finding container 2f2f6d93e8aff45b1e689ebf9cc8210f0a271beed17e334a749bb2522d85d618: Status 404 returned error can't find the container with id 2f2f6d93e8aff45b1e689ebf9cc8210f0a271beed17e334a749bb2522d85d618
Oct 30 01:16:23.585623 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:23.585583    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr" event={"ID":"a627558d-fddd-4135-be1e-b24b424d0479","Type":"ContainerStarted","Data":"6dc1574144ff464579fc22278ad6e3659e0b9e64e4953dfa3d6d4cd3dc4ee23e"}
Oct 30 01:16:23.586136 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:23.585646    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr" event={"ID":"a627558d-fddd-4135-be1e-b24b424d0479","Type":"ContainerStarted","Data":"2f2f6d93e8aff45b1e689ebf9cc8210f0a271beed17e334a749bb2522d85d618"}
Oct 30 01:16:23.586136 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:23.585673    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr"
Oct 30 01:16:23.633037 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:23.632970    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr" podStartSLOduration=1.6329548379999999 podStartE2EDuration="1.632954838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:23.62823125 +0000 UTC m=+2477.485139005" watchObservedRunningTime="2026-10-30 01:16:23.632954838 +0000 UTC m=+2477.489862592"
Oct 30 01:16:26.610631 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:26.610598    6099 generic.go:356] "Generic (PLEG): container finished" podID="87d1ee54-2797-4936-a3a3-44624f7c15c8" containerID="3bf85cbe38b97a8dec59f3de6329addc6f202683d89930fd17bdf53370a74835" exitCode=0
Oct 30 01:16:26.611103 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:26.610675    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9" event={"ID":"87d1ee54-2797-4936-a3a3-44624f7c15c8","Type":"ContainerDied","Data":"3bf85cbe38b97a8dec59f3de6329addc6f202683d89930fd17bdf53370a74835"}
Oct 30 01:16:26.611182 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:26.611128    6099 scope.go:122] "RemoveContainer" containerID="3bf85cbe38b97a8dec59f3de6329addc6f202683d89930fd17bdf53370a74835"
Oct 30 01:16:26.613080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:26.613048    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:16:26.613179 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:26.613119    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:16:26.613179 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:26.613154    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:16:27.622332 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:27.622299    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9" event={"ID":"87d1ee54-2797-4936-a3a3-44624f7c15c8","Type":"ContainerStarted","Data":"4d32b59f61aae37df017bebcaf999f2e90c2ba1f9cc84cd34cbd121a7aa756e8"}
Oct 30 01:16:27.815172 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:27.815139    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:16:27.816001 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:27.815973    6099 patch_prober.go:29] interesting pod/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Oct 30 01:16:27.816120 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:27.816030    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9" podUID="87d1ee54-2797-4936-a3a3-44624f7c15c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Oct 30 01:16:28.815888 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:28.815852    6099 patch_prober.go:29] interesting pod/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Oct 30 01:16:28.816388 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:28.815907    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9" podUID="87d1ee54-2797-4936-a3a3-44624f7c15c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Oct 30 01:16:29.036274 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:29.036196    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf"]
Oct 30 01:16:29.047343 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:29.047313    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1161126e-lmjxf"]
Oct 30 01:16:29.815645 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:29.815611    6099 patch_prober.go:29] interesting pod/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Oct 30 01:16:29.815859 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:29.815675    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9" podUID="87d1ee54-2797-4936-a3a3-44624f7c15c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Oct 30 01:16:30.264060 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:30.263984    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c36496d-2048-4e23-9e47-63c3fcb78bbf" path="/var/lib/kubelet/pods/6c36496d-2048-4e23-9e47-63c3fcb78bbf/volumes"
Oct 30 01:16:30.814833 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:30.814796    6099 patch_prober.go:29] interesting pod/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Oct 30 01:16:30.815026 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:30.814853    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9" podUID="87d1ee54-2797-4936-a3a3-44624f7c15c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Oct 30 01:16:31.815364 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:31.815275    6099 patch_prober.go:29] interesting pod/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Oct 30 01:16:31.815364 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:31.815334    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9" podUID="87d1ee54-2797-4936-a3a3-44624f7c15c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Oct 30 01:16:32.821183 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:32.821152    6099 patch_prober.go:29] interesting pod/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Oct 30 01:16:32.821763 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:32.821723    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9" podUID="87d1ee54-2797-4936-a3a3-44624f7c15c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Oct 30 01:16:33.608112 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:33.608078    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:16:33.815320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:33.815282    6099 patch_prober.go:29] interesting pod/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Oct 30 01:16:33.815510 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:33.815357    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9" podUID="87d1ee54-2797-4936-a3a3-44624f7c15c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Oct 30 01:16:34.615230 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:34.615196    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr"
Oct 30 01:16:34.820748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:34.820718    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:16:34.821976 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:34.821949    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:16:38.349302 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.349271    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"]
Oct 30 01:16:38.353179 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.353154    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.369251 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.369228    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-95492e-openshift-default-dockercfg-lz5kv\""
Oct 30 01:16:38.388568 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.388536    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"]
Oct 30 01:16:38.450441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.450318    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-data\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.450844 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.450820    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-envoy\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.451032 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.451017    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbcb61d-3921-444a-bcc9-1b8382320f04-istiod-ca-cert\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.451183 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.451169    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-workload-certs\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.451371 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.451352    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zjbs\" (UniqueName: \"kubernetes.io/projected/5bbcb61d-3921-444a-bcc9-1b8382320f04-kube-api-access-4zjbs\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.453303 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.453274    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-podinfo\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.453498 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.453481    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-workload-socket\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.455906 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.455885    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-token\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.458926 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.458905    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-credential-socket\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.561428 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.561117    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-credential-socket\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.561605 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.561512    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-data\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.561605 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.561592    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-envoy\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.561689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.561655    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbcb61d-3921-444a-bcc9-1b8382320f04-istiod-ca-cert\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.561753 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.561693    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-workload-certs\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.561811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.561768    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zjbs\" (UniqueName: \"kubernetes.io/projected/5bbcb61d-3921-444a-bcc9-1b8382320f04-kube-api-access-4zjbs\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.561853 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.561827    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-podinfo\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.561889 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.561853    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-workload-socket\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.562102 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.562063    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-credential-socket\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.562246 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.562112    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-token\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.562374 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.562352    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-workload-certs\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.562628 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.562607    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-data\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.563780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.563735    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-workload-socket\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.564478 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.564452    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbcb61d-3921-444a-bcc9-1b8382320f04-istiod-ca-cert\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.566858 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.566807    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-podinfo\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.574249 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.574226    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-envoy\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.595149 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.595116    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zjbs\" (UniqueName: \"kubernetes.io/projected/5bbcb61d-3921-444a-bcc9-1b8382320f04-kube-api-access-4zjbs\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.596610 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.596584    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-token\") pod \"e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") " pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:38.667857 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:38.667762    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:39.081718 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:39.081621    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"]
Oct 30 01:16:39.083253 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:16:39.083217    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbcb61d_3921_444a_bcc9_1b8382320f04.slice/crio-1c7167ccd889d40e94487b8b534de8f6414478a7a41d96ab9f1656af34d5c87a WatchSource:0}: Error finding container 1c7167ccd889d40e94487b8b534de8f6414478a7a41d96ab9f1656af34d5c87a: Status 404 returned error can't find the container with id 1c7167ccd889d40e94487b8b534de8f6414478a7a41d96ab9f1656af34d5c87a
Oct 30 01:16:39.087864 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:39.087785    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:16:39.087953 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:39.087917    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:16:39.088010 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:39.087969    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:16:39.789745 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:39.787051    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q" event={"ID":"5bbcb61d-3921-444a-bcc9-1b8382320f04","Type":"ContainerStarted","Data":"2167109bfbe4f5c784b335dc1d0b085c551bb3191aa2e85d38052c0654af4d3a"}
Oct 30 01:16:39.789745 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:39.787116    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q" event={"ID":"5bbcb61d-3921-444a-bcc9-1b8382320f04","Type":"ContainerStarted","Data":"1c7167ccd889d40e94487b8b534de8f6414478a7a41d96ab9f1656af34d5c87a"}
Oct 30 01:16:39.841247 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:39.841187    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q" podStartSLOduration=1.8411689679999999 podStartE2EDuration="1.841168968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:39.839815769 +0000 UTC m=+2493.696723524" watchObservedRunningTime="2026-10-30 01:16:39.841168968 +0000 UTC m=+2493.698076724"
Oct 30 01:16:40.237789 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.234611    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l"]
Oct 30 01:16:40.520581 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.520495    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l"
Oct 30 01:16:40.539805 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.539776    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-481249\""
Oct 30 01:16:40.541620 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.541597    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-481249-dockercfg-dh2gn\""
Oct 30 01:16:40.642410 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.642379    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l"]
Oct 30 01:16:40.642693 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.642674    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m"]
Oct 30 01:16:40.651981 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.651956    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m"]
Oct 30 01:16:40.652227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.652215    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m"
Oct 30 01:16:40.671285 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.669950    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:40.672425 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.672402    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Oct 30 01:16:40.674194 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.674168    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q" podUID="5bbcb61d-3921-444a-bcc9-1b8382320f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Oct 30 01:16:40.716035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.716000    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5wnr\" (UniqueName: \"kubernetes.io/projected/b2e137cc-9cf8-4212-a7cc-0759867fd7ea-kube-api-access-z5wnr\") pod \"payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l\" (UID: \"b2e137cc-9cf8-4212-a7cc-0759867fd7ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l"
Oct 30 01:16:40.716208 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.716129    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e137cc-9cf8-4212-a7cc-0759867fd7ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l\" (UID: \"b2e137cc-9cf8-4212-a7cc-0759867fd7ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l"
Oct 30 01:16:40.817412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.817378    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e137cc-9cf8-4212-a7cc-0759867fd7ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l\" (UID: \"b2e137cc-9cf8-4212-a7cc-0759867fd7ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l"
Oct 30 01:16:40.817913 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.817519    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf54661-aa8e-4d60-9981-16006d11e41a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m\" (UID: \"1bf54661-aa8e-4d60-9981-16006d11e41a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m"
Oct 30 01:16:40.817913 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.817580    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5z2x\" (UniqueName: \"kubernetes.io/projected/1bf54661-aa8e-4d60-9981-16006d11e41a-kube-api-access-b5z2x\") pod \"payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m\" (UID: \"1bf54661-aa8e-4d60-9981-16006d11e41a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m"
Oct 30 01:16:40.818113 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.818088    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5wnr\" (UniqueName: \"kubernetes.io/projected/b2e137cc-9cf8-4212-a7cc-0759867fd7ea-kube-api-access-z5wnr\") pod \"payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l\" (UID: \"b2e137cc-9cf8-4212-a7cc-0759867fd7ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l"
Oct 30 01:16:40.818180 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.818135    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e137cc-9cf8-4212-a7cc-0759867fd7ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l\" (UID: \"b2e137cc-9cf8-4212-a7cc-0759867fd7ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l"
Oct 30 01:16:40.841157 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.841131    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5wnr\" (UniqueName: \"kubernetes.io/projected/b2e137cc-9cf8-4212-a7cc-0759867fd7ea-kube-api-access-z5wnr\") pod \"payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l\" (UID: \"b2e137cc-9cf8-4212-a7cc-0759867fd7ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l"
Oct 30 01:16:40.919893 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.919856    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf54661-aa8e-4d60-9981-16006d11e41a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m\" (UID: \"1bf54661-aa8e-4d60-9981-16006d11e41a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m"
Oct 30 01:16:40.920105 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.919993    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5z2x\" (UniqueName: \"kubernetes.io/projected/1bf54661-aa8e-4d60-9981-16006d11e41a-kube-api-access-b5z2x\") pod \"payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m\" (UID: \"1bf54661-aa8e-4d60-9981-16006d11e41a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m"
Oct 30 01:16:40.920790 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.920765    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf54661-aa8e-4d60-9981-16006d11e41a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m\" (UID: \"1bf54661-aa8e-4d60-9981-16006d11e41a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m"
Oct 30 01:16:40.943078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.943054    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5z2x\" (UniqueName: \"kubernetes.io/projected/1bf54661-aa8e-4d60-9981-16006d11e41a-kube-api-access-b5z2x\") pod \"payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m\" (UID: \"1bf54661-aa8e-4d60-9981-16006d11e41a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m"
Oct 30 01:16:40.950079 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.950062    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l"
Oct 30 01:16:40.977050 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:40.977014    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m"
Oct 30 01:16:41.059484 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:41.059251    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh"]
Oct 30 01:16:41.068776 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:41.064855    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e065abe8-hxnmh"]
Oct 30 01:16:41.390770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:41.389789    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l"]
Oct 30 01:16:41.390770 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:16:41.389824    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e137cc_9cf8_4212_a7cc_0759867fd7ea.slice/crio-a404845e32c57cec94928f27d1060faff458f0f34b1320336eb5e8913537c68e WatchSource:0}: Error finding container a404845e32c57cec94928f27d1060faff458f0f34b1320336eb5e8913537c68e: Status 404 returned error can't find the container with id a404845e32c57cec94928f27d1060faff458f0f34b1320336eb5e8913537c68e
Oct 30 01:16:41.406382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:41.406312    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m"]
Oct 30 01:16:41.409699 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:16:41.409674    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf54661_aa8e_4d60_9981_16006d11e41a.slice/crio-74e0d62664712f7e3605bd23265a778ab1ee47c1b8ad6628c8d02f2357e2c5cd WatchSource:0}: Error finding container 74e0d62664712f7e3605bd23265a778ab1ee47c1b8ad6628c8d02f2357e2c5cd: Status 404 returned error can't find the container with id 74e0d62664712f7e3605bd23265a778ab1ee47c1b8ad6628c8d02f2357e2c5cd
Oct 30 01:16:41.689974 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:41.686810    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Oct 30 01:16:41.689974 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:41.686870    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q" podUID="5bbcb61d-3921-444a-bcc9-1b8382320f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Oct 30 01:16:41.810091 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:41.810037    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l" event={"ID":"b2e137cc-9cf8-4212-a7cc-0759867fd7ea","Type":"ContainerStarted","Data":"a404845e32c57cec94928f27d1060faff458f0f34b1320336eb5e8913537c68e"}
Oct 30 01:16:41.811973 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:41.811929    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m" event={"ID":"1bf54661-aa8e-4d60-9981-16006d11e41a","Type":"ContainerStarted","Data":"74e0d62664712f7e3605bd23265a778ab1ee47c1b8ad6628c8d02f2357e2c5cd"}
Oct 30 01:16:42.302961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:42.302929    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c5e63e-5254-4961-94b2-4098db32e569" path="/var/lib/kubelet/pods/27c5e63e-5254-4961-94b2-4098db32e569/volumes"
Oct 30 01:16:42.680114 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:42.669073    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Oct 30 01:16:42.680114 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:42.669135    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q" podUID="5bbcb61d-3921-444a-bcc9-1b8382320f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Oct 30 01:16:43.688731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:43.684776    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Oct 30 01:16:43.688731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:43.684836    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q" podUID="5bbcb61d-3921-444a-bcc9-1b8382320f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Oct 30 01:16:44.708790 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:44.683355    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Oct 30 01:16:44.756730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:44.743789    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q" podUID="5bbcb61d-3921-444a-bcc9-1b8382320f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Oct 30 01:16:45.058793 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:45.058759    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m"
Oct 30 01:16:45.111768 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:45.111684    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m" podStartSLOduration=2.379151563 podStartE2EDuration="5.111667906s" totalImagesPullingTime="2.732516343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:45.105997572 +0000 UTC m=+2498.962905324" watchObservedRunningTime="2026-10-30 01:16:45.111667906 +0000 UTC m=+2498.968575662"
Oct 30 01:16:45.674727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:45.674056    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Oct 30 01:16:45.674727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:45.674122    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q" podUID="5bbcb61d-3921-444a-bcc9-1b8382320f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Oct 30 01:16:46.073362 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:46.073322    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l" event={"ID":"b2e137cc-9cf8-4212-a7cc-0759867fd7ea","Type":"ContainerStarted","Data":"bb217a75db40af03cb0e0a4df0bf97b5e7d8ebdbd495c7f1253b48e09b21911e"}
Oct 30 01:16:46.073985 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:46.073961    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l"
Oct 30 01:16:46.075471 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:46.075442    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m" event={"ID":"1bf54661-aa8e-4d60-9981-16006d11e41a","Type":"ContainerStarted","Data":"7a087ac5a4d45b1e9c57b67368d89b4a2d5ca0de6494392955457d492d6d448e"}
Oct 30 01:16:46.137317 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:46.133766    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l" podStartSLOduration=3.403990549 podStartE2EDuration="6.133741444s" totalImagesPullingTime="2.729750895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:46.122504429 +0000 UTC m=+2499.979412187" watchObservedRunningTime="2026-10-30 01:16:46.133741444 +0000 UTC m=+2499.990649192"
Oct 30 01:16:46.734503 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:16:46.734464    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:16:47.043726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:47.040587    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:47.043726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:47.042497    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:47.043959 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:47.043819    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:16:57.092155 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:57.092124    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l"
Oct 30 01:16:57.092738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:16:57.092288    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m"
Oct 30 01:17:01.996846 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:01.996807    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"]
Oct 30 01:17:14.774913 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:14.774883    6099 scope.go:122] "RemoveContainer" containerID="9dcdb469d40edb9167ed8f5786ad507539e9e1a2eadcf70accc6f74b1662534b"
Oct 30 01:17:16.632586 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:17:16.632548    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:17:17.034574 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.034474    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn" podUID="e36adae7-03ed-4647-9410-5b4bceafa593" containerName="main" containerID="cri-o://0dffec5ca437a3adfb0d19315491859e85503c5bffc793915778c8cea386157b" gracePeriod=45
Oct 30 01:17:17.333004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.332981    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:17:17.407411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.407381    6099 generic.go:356] "Generic (PLEG): container finished" podID="e36adae7-03ed-4647-9410-5b4bceafa593" containerID="0dffec5ca437a3adfb0d19315491859e85503c5bffc793915778c8cea386157b" exitCode=0
Oct 30 01:17:17.407605 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.407426    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"
Oct 30 01:17:17.407605 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.407480    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn" event={"ID":"e36adae7-03ed-4647-9410-5b4bceafa593","Type":"ContainerDied","Data":"0dffec5ca437a3adfb0d19315491859e85503c5bffc793915778c8cea386157b"}
Oct 30 01:17:17.407605 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.407543    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn" event={"ID":"e36adae7-03ed-4647-9410-5b4bceafa593","Type":"ContainerDied","Data":"6796a70404cb700bd4cda68be2f0644d56ab5b837f8ddd7ec6a66e7e2e878bf0"}
Oct 30 01:17:17.407605 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.407569    6099 scope.go:122] "RemoveContainer" containerID="0dffec5ca437a3adfb0d19315491859e85503c5bffc793915778c8cea386157b"
Oct 30 01:17:17.414579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.414551    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-tmp-dir\") pod \"e36adae7-03ed-4647-9410-5b4bceafa593\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") "
Oct 30 01:17:17.414848 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.414827    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e36adae7-03ed-4647-9410-5b4bceafa593-tls-certs\" (UniqueName: \"kubernetes.io/secret/e36adae7-03ed-4647-9410-5b4bceafa593-tls-certs\") pod \"e36adae7-03ed-4647-9410-5b4bceafa593\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") "
Oct 30 01:17:17.415080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.415062    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-home\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-home\") pod \"e36adae7-03ed-4647-9410-5b4bceafa593\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") "
Oct 30 01:17:17.416548 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.416524    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e36adae7-03ed-4647-9410-5b4bceafa593-kube-api-access-9r4mj\" (UniqueName: \"kubernetes.io/projected/e36adae7-03ed-4647-9410-5b4bceafa593-kube-api-access-9r4mj\") pod \"e36adae7-03ed-4647-9410-5b4bceafa593\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") "
Oct 30 01:17:17.417556 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.417534    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-model-cache\") pod \"e36adae7-03ed-4647-9410-5b4bceafa593\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") "
Oct 30 01:17:17.418003 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.417980    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-dshm\") pod \"e36adae7-03ed-4647-9410-5b4bceafa593\" (UID: \"e36adae7-03ed-4647-9410-5b4bceafa593\") "
Oct 30 01:17:17.430831 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.430419    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-home" pod "e36adae7-03ed-4647-9410-5b4bceafa593" (UID: "e36adae7-03ed-4647-9410-5b4bceafa593"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:17.430831 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.430351    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-tmp-dir" pod "e36adae7-03ed-4647-9410-5b4bceafa593" (UID: "e36adae7-03ed-4647-9410-5b4bceafa593"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:17.435525 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.435384    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-model-cache" pod "e36adae7-03ed-4647-9410-5b4bceafa593" (UID: "e36adae7-03ed-4647-9410-5b4bceafa593"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:17.472108 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.460014    6099 scope.go:122] "RemoveContainer" containerID="0dffec5ca437a3adfb0d19315491859e85503c5bffc793915778c8cea386157b"
Oct 30 01:17:17.472108 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:17:17.460496    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dffec5ca437a3adfb0d19315491859e85503c5bffc793915778c8cea386157b\": container with ID starting with 0dffec5ca437a3adfb0d19315491859e85503c5bffc793915778c8cea386157b not found: ID does not exist" containerID="0dffec5ca437a3adfb0d19315491859e85503c5bffc793915778c8cea386157b"
Oct 30 01:17:17.472108 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.460555    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dffec5ca437a3adfb0d19315491859e85503c5bffc793915778c8cea386157b"} err="failed to get container status \"0dffec5ca437a3adfb0d19315491859e85503c5bffc793915778c8cea386157b\": rpc error: code = NotFound desc = could not find container \"0dffec5ca437a3adfb0d19315491859e85503c5bffc793915778c8cea386157b\": container with ID starting with 0dffec5ca437a3adfb0d19315491859e85503c5bffc793915778c8cea386157b not found: ID does not exist"
Oct 30 01:17:17.478576 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.478533    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-dshm" pod "e36adae7-03ed-4647-9410-5b4bceafa593" (UID: "e36adae7-03ed-4647-9410-5b4bceafa593"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:17.480298 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.480241    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e36adae7-03ed-4647-9410-5b4bceafa593-kube-api-access-9r4mj" pod "e36adae7-03ed-4647-9410-5b4bceafa593" (UID: "e36adae7-03ed-4647-9410-5b4bceafa593"). InnerVolumeSpecName "kube-api-access-9r4mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:17.482777 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.482699    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e36adae7-03ed-4647-9410-5b4bceafa593-tls-certs" pod "e36adae7-03ed-4647-9410-5b4bceafa593" (UID: "e36adae7-03ed-4647-9410-5b4bceafa593"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:17.529614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.529559    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:17:17.529614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.529590    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:17:17.529614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.529599    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:17:17.529614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.529611    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e36adae7-03ed-4647-9410-5b4bceafa593-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:17:17.529614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.529623    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e36adae7-03ed-4647-9410-5b4bceafa593-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:17:17.530099 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.529635    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r4mj\" (UniqueName: \"kubernetes.io/projected/e36adae7-03ed-4647-9410-5b4bceafa593-kube-api-access-9r4mj\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:17:17.765690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.765657    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"]
Oct 30 01:17:17.777131 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:17.777105    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn"]
Oct 30 01:17:18.217783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:18.217689    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-95492e/sub-test-model-95492e-kserve-84dc465d9c-lgcxn" podUID="e36adae7-03ed-4647-9410-5b4bceafa593" containerName="main" probeResult="failure" output="Get \"https://10.134.1.17:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:17:18.268897 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:18.268858    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e36adae7-03ed-4647-9410-5b4bceafa593" path="/var/lib/kubelet/pods/e36adae7-03ed-4647-9410-5b4bceafa593/volumes"
Oct 30 01:17:38.576730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:17:38.574630    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:17:46.631198 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:17:46.631161    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:18:06.582572 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:06.582531    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"]
Oct 30 01:18:14.805808 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:14.805775    6099 scope.go:122] "RemoveContainer" containerID="b29849b7249a3bf65d30024f5e2b61fb7ddba9c5eb6443ca5e2fa1394e2d6154"
Oct 30 01:18:14.817760 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:14.817740    6099 scope.go:122] "RemoveContainer" containerID="08e21d7d3c5cbf726abbb3f3ad2d538725a2994c45a3abaf23016330b905ca8a"
Oct 30 01:18:16.632311 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:18:16.632274    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:18:18.286961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:18.286922    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4"]
Oct 30 01:18:18.319334 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:18.319298    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p"]
Oct 30 01:18:21.605654 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:21.605606    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26" podUID="6b163830-1b90-4bcc-8238-a08bc5b9cb7e" containerName="main" containerID="cri-o://ed2ee8dede2dbbd090295aad1b3c7d70ec78c3e6891de2866733491a260519c2" gracePeriod=45
Oct 30 01:18:21.902127 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:21.902103    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:18:22.032876 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.032847    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-model-cache\") pod \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") "
Oct 30 01:18:22.033060 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.032979    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-home\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-home\") pod \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") "
Oct 30 01:18:22.033060 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.033007    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-dshm\") pod \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") "
Oct 30 01:18:22.033172 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.033135    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-model-cache" pod "6b163830-1b90-4bcc-8238-a08bc5b9cb7e" (UID: "6b163830-1b90-4bcc-8238-a08bc5b9cb7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:22.033275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.033256    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-home" pod "6b163830-1b90-4bcc-8238-a08bc5b9cb7e" (UID: "6b163830-1b90-4bcc-8238-a08bc5b9cb7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:22.033325 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.033308    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-tls-certs\") pod \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") "
Oct 30 01:18:22.033391 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.033376    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-kube-api-access-tt8mm\" (UniqueName: \"kubernetes.io/projected/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-kube-api-access-tt8mm\") pod \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") "
Oct 30 01:18:22.033451 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.033435    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-tmp-dir\") pod \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\" (UID: \"6b163830-1b90-4bcc-8238-a08bc5b9cb7e\") "
Oct 30 01:18:22.033820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.033765    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-tmp-dir" pod "6b163830-1b90-4bcc-8238-a08bc5b9cb7e" (UID: "6b163830-1b90-4bcc-8238-a08bc5b9cb7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:22.034251 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.034234    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:18:22.034326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.034253    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:18:22.034326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.034262    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:18:22.035871 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.035847    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-dshm" pod "6b163830-1b90-4bcc-8238-a08bc5b9cb7e" (UID: "6b163830-1b90-4bcc-8238-a08bc5b9cb7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:22.036001 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.035880    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-tls-certs" pod "6b163830-1b90-4bcc-8238-a08bc5b9cb7e" (UID: "6b163830-1b90-4bcc-8238-a08bc5b9cb7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:22.036001 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.035930    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-kube-api-access-tt8mm" pod "6b163830-1b90-4bcc-8238-a08bc5b9cb7e" (UID: "6b163830-1b90-4bcc-8238-a08bc5b9cb7e"). InnerVolumeSpecName "kube-api-access-tt8mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:22.135483 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.135402    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:18:22.135483 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.135431    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt8mm\" (UniqueName: \"kubernetes.io/projected/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-kube-api-access-tt8mm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:18:22.135483 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.135444    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b163830-1b90-4bcc-8238-a08bc5b9cb7e-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:18:22.147425 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.147390    6099 generic.go:356] "Generic (PLEG): container finished" podID="6b163830-1b90-4bcc-8238-a08bc5b9cb7e" containerID="ed2ee8dede2dbbd090295aad1b3c7d70ec78c3e6891de2866733491a260519c2" exitCode=0
Oct 30 01:18:22.147581 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.147436    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"
Oct 30 01:18:22.147581 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.147476    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26" event={"ID":"6b163830-1b90-4bcc-8238-a08bc5b9cb7e","Type":"ContainerDied","Data":"ed2ee8dede2dbbd090295aad1b3c7d70ec78c3e6891de2866733491a260519c2"}
Oct 30 01:18:22.147581 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.147522    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26" event={"ID":"6b163830-1b90-4bcc-8238-a08bc5b9cb7e","Type":"ContainerDied","Data":"46a52887557b7a51c568cff4c9c1a6424dc182ea3c4837659bb1bc0cc2c161e8"}
Oct 30 01:18:22.147581 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.147540    6099 scope.go:122] "RemoveContainer" containerID="ed2ee8dede2dbbd090295aad1b3c7d70ec78c3e6891de2866733491a260519c2"
Oct 30 01:18:22.159254 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.159236    6099 scope.go:122] "RemoveContainer" containerID="ed2ee8dede2dbbd090295aad1b3c7d70ec78c3e6891de2866733491a260519c2"
Oct 30 01:18:22.159493 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:18:22.159473    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed2ee8dede2dbbd090295aad1b3c7d70ec78c3e6891de2866733491a260519c2\": container with ID starting with ed2ee8dede2dbbd090295aad1b3c7d70ec78c3e6891de2866733491a260519c2 not found: ID does not exist" containerID="ed2ee8dede2dbbd090295aad1b3c7d70ec78c3e6891de2866733491a260519c2"
Oct 30 01:18:22.159552 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.159502    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2ee8dede2dbbd090295aad1b3c7d70ec78c3e6891de2866733491a260519c2"} err="failed to get container status \"ed2ee8dede2dbbd090295aad1b3c7d70ec78c3e6891de2866733491a260519c2\": rpc error: code = NotFound desc = could not find container \"ed2ee8dede2dbbd090295aad1b3c7d70ec78c3e6891de2866733491a260519c2\": container with ID starting with ed2ee8dede2dbbd090295aad1b3c7d70ec78c3e6891de2866733491a260519c2 not found: ID does not exist"
Oct 30 01:18:22.201304 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.201280    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"]
Oct 30 01:18:22.204126 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.204100    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-320045/auth-test-model-320045-kserve-54468b6b85-8zl26"]
Oct 30 01:18:22.264224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:22.264194    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b163830-1b90-4bcc-8238-a08bc5b9cb7e" path="/var/lib/kubelet/pods/6b163830-1b90-4bcc-8238-a08bc5b9cb7e/volumes"
Oct 30 01:18:23.287918 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:23.287878    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4" podUID="d3bfb222-a8f8-47bc-a3cd-a62716211f0e" containerName="payload-processing" containerID="cri-o://5b9a5538abb8f89f82732a9e0584c1f58642ae673a1c6a509b540d16d0b1c88b" gracePeriod=25
Oct 30 01:18:23.319989 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:23.319939    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p" podUID="7ed5048f-9425-41d3-8d13-7b149709bf3a" containerName="payload-pre-processing" containerID="cri-o://7ab173004a8dbd025bd04cd462f30a89d5e467fca6fefa00af92004de97159a1" gracePeriod=25
Oct 30 01:18:26.514995 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:26.514956    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p" podUID="7ed5048f-9425-41d3-8d13-7b149709bf3a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:18:26.515469 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:26.515018    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4" podUID="d3bfb222-a8f8-47bc-a3cd-a62716211f0e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:18:28.809493 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:28.809444    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:18:36.515595 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:36.515557    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p" podUID="7ed5048f-9425-41d3-8d13-7b149709bf3a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:18:36.516059 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:36.515634    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4" podUID="d3bfb222-a8f8-47bc-a3cd-a62716211f0e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:18:43.446771 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.446738    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p" event={"ID":"7ed5048f-9425-41d3-8d13-7b149709bf3a","Type":"ContainerDied","Data":"7ab173004a8dbd025bd04cd462f30a89d5e467fca6fefa00af92004de97159a1"}
Oct 30 01:18:43.447135 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.446746    6099 generic.go:356] "Generic (PLEG): container finished" podID="7ed5048f-9425-41d3-8d13-7b149709bf3a" containerID="7ab173004a8dbd025bd04cd462f30a89d5e467fca6fefa00af92004de97159a1" exitCode=0
Oct 30 01:18:43.449054 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.449008    6099 generic.go:356] "Generic (PLEG): container finished" podID="d3bfb222-a8f8-47bc-a3cd-a62716211f0e" containerID="5b9a5538abb8f89f82732a9e0584c1f58642ae673a1c6a509b540d16d0b1c88b" exitCode=0
Oct 30 01:18:43.449179 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.449080    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4" event={"ID":"d3bfb222-a8f8-47bc-a3cd-a62716211f0e","Type":"ContainerDied","Data":"5b9a5538abb8f89f82732a9e0584c1f58642ae673a1c6a509b540d16d0b1c88b"}
Oct 30 01:18:43.626628 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.626603    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4"
Oct 30 01:18:43.698902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.688794    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf"]
Oct 30 01:18:43.698902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.690484    6099 cpu_manager.go:414] "containerMap: removing container" podUID="6b163830-1b90-4bcc-8238-a08bc5b9cb7e" containerName="main"
Oct 30 01:18:43.698902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.690505    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b163830-1b90-4bcc-8238-a08bc5b9cb7e" containerName="main"
Oct 30 01:18:43.698902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.690588    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e36adae7-03ed-4647-9410-5b4bceafa593" containerName="main"
Oct 30 01:18:43.698902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.690595    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e36adae7-03ed-4647-9410-5b4bceafa593" containerName="main"
Oct 30 01:18:43.698902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.690625    6099 cpu_manager.go:414] "containerMap: removing container" podUID="d3bfb222-a8f8-47bc-a3cd-a62716211f0e" containerName="payload-processing"
Oct 30 01:18:43.698902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.690633    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3bfb222-a8f8-47bc-a3cd-a62716211f0e" containerName="payload-processing"
Oct 30 01:18:43.698902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.690857    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e36adae7-03ed-4647-9410-5b4bceafa593" containerName="main"
Oct 30 01:18:43.698902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.690877    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3bfb222-a8f8-47bc-a3cd-a62716211f0e" containerName="payload-processing"
Oct 30 01:18:43.698902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.690897    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b163830-1b90-4bcc-8238-a08bc5b9cb7e" containerName="main"
Oct 30 01:18:43.737940 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.730853    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3bfb222-a8f8-47bc-a3cd-a62716211f0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3bfb222-a8f8-47bc-a3cd-a62716211f0e-plugins-config-volume\") pod \"d3bfb222-a8f8-47bc-a3cd-a62716211f0e\" (UID: \"d3bfb222-a8f8-47bc-a3cd-a62716211f0e\") "
Oct 30 01:18:43.737940 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.731016    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3bfb222-a8f8-47bc-a3cd-a62716211f0e-kube-api-access-24hm2\" (UniqueName: \"kubernetes.io/projected/d3bfb222-a8f8-47bc-a3cd-a62716211f0e-kube-api-access-24hm2\") pod \"d3bfb222-a8f8-47bc-a3cd-a62716211f0e\" (UID: \"d3bfb222-a8f8-47bc-a3cd-a62716211f0e\") "
Oct 30 01:18:43.737940 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.732142    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3bfb222-a8f8-47bc-a3cd-a62716211f0e-plugins-config-volume" pod "d3bfb222-a8f8-47bc-a3cd-a62716211f0e" (UID: "d3bfb222-a8f8-47bc-a3cd-a62716211f0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:43.737940 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.736976    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p"
Oct 30 01:18:43.737940 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.737444    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3bfb222-a8f8-47bc-a3cd-a62716211f0e-kube-api-access-24hm2" pod "d3bfb222-a8f8-47bc-a3cd-a62716211f0e" (UID: "d3bfb222-a8f8-47bc-a3cd-a62716211f0e"). InnerVolumeSpecName "kube-api-access-24hm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:43.743242 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.743204    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf"]
Oct 30 01:18:43.743434 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.743304    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf"
Oct 30 01:18:43.762036 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.761222    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d1ae36-dockercfg-wbvrw\""
Oct 30 01:18:43.762036 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.761455    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d1ae36\""
Oct 30 01:18:43.834655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.834002    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxk9q\" (UniqueName: \"kubernetes.io/projected/c3dec5ee-7aba-4409-b10d-bed8d78c1df3-kube-api-access-zxk9q\") pod \"payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf\" (UID: \"c3dec5ee-7aba-4409-b10d-bed8d78c1df3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf"
Oct 30 01:18:43.834655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.834186    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3dec5ee-7aba-4409-b10d-bed8d78c1df3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf\" (UID: \"c3dec5ee-7aba-4409-b10d-bed8d78c1df3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf"
Oct 30 01:18:43.834655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.834389    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3bfb222-a8f8-47bc-a3cd-a62716211f0e-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:18:43.834655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.834407    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24hm2\" (UniqueName: \"kubernetes.io/projected/d3bfb222-a8f8-47bc-a3cd-a62716211f0e-kube-api-access-24hm2\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:18:43.935137 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.935107    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed5048f-9425-41d3-8d13-7b149709bf3a-kube-api-access-wh97l\" (UniqueName: \"kubernetes.io/projected/7ed5048f-9425-41d3-8d13-7b149709bf3a-kube-api-access-wh97l\") pod \"7ed5048f-9425-41d3-8d13-7b149709bf3a\" (UID: \"7ed5048f-9425-41d3-8d13-7b149709bf3a\") "
Oct 30 01:18:43.935303 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.935231    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed5048f-9425-41d3-8d13-7b149709bf3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed5048f-9425-41d3-8d13-7b149709bf3a-plugins-config-volume\") pod \"7ed5048f-9425-41d3-8d13-7b149709bf3a\" (UID: \"7ed5048f-9425-41d3-8d13-7b149709bf3a\") "
Oct 30 01:18:43.935577 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.935560    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxk9q\" (UniqueName: \"kubernetes.io/projected/c3dec5ee-7aba-4409-b10d-bed8d78c1df3-kube-api-access-zxk9q\") pod \"payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf\" (UID: \"c3dec5ee-7aba-4409-b10d-bed8d78c1df3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf"
Oct 30 01:18:43.935666 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.935568    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed5048f-9425-41d3-8d13-7b149709bf3a-plugins-config-volume" pod "7ed5048f-9425-41d3-8d13-7b149709bf3a" (UID: "7ed5048f-9425-41d3-8d13-7b149709bf3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:43.935749 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.935662    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3dec5ee-7aba-4409-b10d-bed8d78c1df3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf\" (UID: \"c3dec5ee-7aba-4409-b10d-bed8d78c1df3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf"
Oct 30 01:18:43.935974 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.935952    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed5048f-9425-41d3-8d13-7b149709bf3a-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:18:43.936399 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.936368    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3dec5ee-7aba-4409-b10d-bed8d78c1df3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf\" (UID: \"c3dec5ee-7aba-4409-b10d-bed8d78c1df3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf"
Oct 30 01:18:43.937390 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.937367    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed5048f-9425-41d3-8d13-7b149709bf3a-kube-api-access-wh97l" pod "7ed5048f-9425-41d3-8d13-7b149709bf3a" (UID: "7ed5048f-9425-41d3-8d13-7b149709bf3a"). InnerVolumeSpecName "kube-api-access-wh97l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:43.958632 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:43.958610    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxk9q\" (UniqueName: \"kubernetes.io/projected/c3dec5ee-7aba-4409-b10d-bed8d78c1df3-kube-api-access-zxk9q\") pod \"payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf\" (UID: \"c3dec5ee-7aba-4409-b10d-bed8d78c1df3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf"
Oct 30 01:18:44.037575 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:44.037548    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh97l\" (UniqueName: \"kubernetes.io/projected/7ed5048f-9425-41d3-8d13-7b149709bf3a-kube-api-access-wh97l\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:18:44.073270 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:44.073244    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf"
Oct 30 01:18:44.378603 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:18:44.378560    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3dec5ee_7aba_4409_b10d_bed8d78c1df3.slice/crio-23ce837a7f5cfb4c14814b2a46a2d66985eb0a1ec6c89335d8a0d98e6828f8a7 WatchSource:0}: Error finding container 23ce837a7f5cfb4c14814b2a46a2d66985eb0a1ec6c89335d8a0d98e6828f8a7: Status 404 returned error can't find the container with id 23ce837a7f5cfb4c14814b2a46a2d66985eb0a1ec6c89335d8a0d98e6828f8a7
Oct 30 01:18:44.380317 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:44.380289    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf"]
Oct 30 01:18:44.460164 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:44.460133    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf" event={"ID":"c3dec5ee-7aba-4409-b10d-bed8d78c1df3","Type":"ContainerStarted","Data":"23ce837a7f5cfb4c14814b2a46a2d66985eb0a1ec6c89335d8a0d98e6828f8a7"}
Oct 30 01:18:44.462321 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:44.462299    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p"
Oct 30 01:18:44.462492 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:44.462412    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p" event={"ID":"7ed5048f-9425-41d3-8d13-7b149709bf3a","Type":"ContainerDied","Data":"5c7b746397bf9d52af3deefe22ffe616f26a72cf9d253b42e9a83dd44a097a4a"}
Oct 30 01:18:44.462492 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:44.462477    6099 scope.go:122] "RemoveContainer" containerID="7ab173004a8dbd025bd04cd462f30a89d5e467fca6fefa00af92004de97159a1"
Oct 30 01:18:44.465242 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:44.465218    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4" event={"ID":"d3bfb222-a8f8-47bc-a3cd-a62716211f0e","Type":"ContainerDied","Data":"1613f07a0c36d13a3112b609f40ea87687cbae01303be2b29fd584d702785340"}
Oct 30 01:18:44.465394 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:44.465310    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4"
Oct 30 01:18:44.475334 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:44.475302    6099 scope.go:122] "RemoveContainer" containerID="5b9a5538abb8f89f82732a9e0584c1f58642ae673a1c6a509b540d16d0b1c88b"
Oct 30 01:18:44.517409 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:44.517380    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p"]
Oct 30 01:18:44.521416 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:44.521386    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-320045-65b9b58b99-z8z7p"]
Oct 30 01:18:44.554493 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:44.554459    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4"]
Oct 30 01:18:44.558171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:44.558147    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-320045-8557b8498-ttln4"]
Oct 30 01:18:45.477658 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:45.477619    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf" event={"ID":"c3dec5ee-7aba-4409-b10d-bed8d78c1df3","Type":"ContainerStarted","Data":"ee706cccfcba4bfb4c05cad3045032506f88dcdf7b69907b90847d5ade7a340d"}
Oct 30 01:18:45.478176 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:45.477691    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf"
Oct 30 01:18:45.522640 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:45.522573    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf" podStartSLOduration=68.522554823 podStartE2EDuration="1m8.522554823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:45.518161698 +0000 UTC m=+2619.375069454" watchObservedRunningTime="2026-10-30 01:18:45.522554823 +0000 UTC m=+2619.379462587"
Oct 30 01:18:46.264916 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:46.264882    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed5048f-9425-41d3-8d13-7b149709bf3a" path="/var/lib/kubelet/pods/7ed5048f-9425-41d3-8d13-7b149709bf3a/volumes"
Oct 30 01:18:46.265469 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:46.265454    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3bfb222-a8f8-47bc-a3cd-a62716211f0e" path="/var/lib/kubelet/pods/d3bfb222-a8f8-47bc-a3cd-a62716211f0e/volumes"
Oct 30 01:18:46.631694 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:18:46.631659    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:18:56.491581 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:18:56.491552    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf"
Oct 30 01:19:03.236592 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:03.236559    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"]
Oct 30 01:19:03.237033 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:03.236756    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w" podUID="5b07e253-0493-423b-a506-eaebd9952e34" containerName="maas-api" containerID="cri-o://e8a2d8dcad1c5723e706475af6eafee87f8d680979c575f13b0976abadb872f0" gracePeriod=30
Oct 30 01:19:03.823068 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:03.823036    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"
Oct 30 01:19:03.857529 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:03.857483    6099 generic.go:356] "Generic (PLEG): container finished" podID="5b07e253-0493-423b-a506-eaebd9952e34" containerID="e8a2d8dcad1c5723e706475af6eafee87f8d680979c575f13b0976abadb872f0" exitCode=0
Oct 30 01:19:03.857945 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:03.857919    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w" event={"ID":"5b07e253-0493-423b-a506-eaebd9952e34","Type":"ContainerDied","Data":"e8a2d8dcad1c5723e706475af6eafee87f8d680979c575f13b0976abadb872f0"}
Oct 30 01:19:03.858149 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:03.858125    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w" event={"ID":"5b07e253-0493-423b-a506-eaebd9952e34","Type":"ContainerDied","Data":"4201ed584055338bbc72b72bec7c89b4988f82ab35d9c40f532a1e21a7d068d3"}
Oct 30 01:19:03.858284 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:03.858270    6099 scope.go:122] "RemoveContainer" containerID="e8a2d8dcad1c5723e706475af6eafee87f8d680979c575f13b0976abadb872f0"
Oct 30 01:19:03.932118 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:03.932081    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b07e253-0493-423b-a506-eaebd9952e34-kube-api-access-lp94b\" (UniqueName: \"kubernetes.io/projected/5b07e253-0493-423b-a506-eaebd9952e34-kube-api-access-lp94b\") pod \"5b07e253-0493-423b-a506-eaebd9952e34\" (UID: \"5b07e253-0493-423b-a506-eaebd9952e34\") "
Oct 30 01:19:03.943882 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:03.943847    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b07e253-0493-423b-a506-eaebd9952e34-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b07e253-0493-423b-a506-eaebd9952e34-maas-api-tls\") pod \"5b07e253-0493-423b-a506-eaebd9952e34\" (UID: \"5b07e253-0493-423b-a506-eaebd9952e34\") "
Oct 30 01:19:03.944146 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:03.944129    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b07e253-0493-423b-a506-eaebd9952e34-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b07e253-0493-423b-a506-eaebd9952e34-metrics-certs\") pod \"5b07e253-0493-423b-a506-eaebd9952e34\" (UID: \"5b07e253-0493-423b-a506-eaebd9952e34\") "
Oct 30 01:19:03.944440 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:03.944423    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b07e253-0493-423b-a506-eaebd9952e34-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b07e253-0493-423b-a506-eaebd9952e34-openshift-service-ca\") pod \"5b07e253-0493-423b-a506-eaebd9952e34\" (UID: \"5b07e253-0493-423b-a506-eaebd9952e34\") "
Oct 30 01:19:03.964103 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:03.947216    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b07e253-0493-423b-a506-eaebd9952e34-kube-api-access-lp94b" pod "5b07e253-0493-423b-a506-eaebd9952e34" (UID: "5b07e253-0493-423b-a506-eaebd9952e34"). InnerVolumeSpecName "kube-api-access-lp94b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:03.964333 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:03.961048    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b07e253-0493-423b-a506-eaebd9952e34-openshift-service-ca" pod "5b07e253-0493-423b-a506-eaebd9952e34" (UID: "5b07e253-0493-423b-a506-eaebd9952e34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:03.976784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:03.975165    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"]
Oct 30 01:19:04.039460 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.039410    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b07e253-0493-423b-a506-eaebd9952e34-maas-api-tls" pod "5b07e253-0493-423b-a506-eaebd9952e34" (UID: "5b07e253-0493-423b-a506-eaebd9952e34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:19:04.039655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.039629    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b07e253-0493-423b-a506-eaebd9952e34-metrics-certs" pod "5b07e253-0493-423b-a506-eaebd9952e34" (UID: "5b07e253-0493-423b-a506-eaebd9952e34"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:19:04.063118 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.063007    6099 cpu_manager.go:414] "containerMap: removing container" podUID="7ed5048f-9425-41d3-8d13-7b149709bf3a" containerName="payload-pre-processing"
Oct 30 01:19:04.063118 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.063037    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed5048f-9425-41d3-8d13-7b149709bf3a" containerName="payload-pre-processing"
Oct 30 01:19:04.063118 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.063087    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5b07e253-0493-423b-a506-eaebd9952e34" containerName="maas-api"
Oct 30 01:19:04.063118 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.063100    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b07e253-0493-423b-a506-eaebd9952e34" containerName="maas-api"
Oct 30 01:19:04.063604 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.063582    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed5048f-9425-41d3-8d13-7b149709bf3a" containerName="payload-pre-processing"
Oct 30 01:19:04.063672 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.063618    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b07e253-0493-423b-a506-eaebd9952e34" containerName="maas-api"
Oct 30 01:19:04.072064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.071259    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp94b\" (UniqueName: \"kubernetes.io/projected/5b07e253-0493-423b-a506-eaebd9952e34-kube-api-access-lp94b\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:04.072064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.071325    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b07e253-0493-423b-a506-eaebd9952e34-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:04.072064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.071350    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b07e253-0493-423b-a506-eaebd9952e34-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:04.072064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.071364    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b07e253-0493-423b-a506-eaebd9952e34-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:04.073054 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.072999    6099 scope.go:122] "RemoveContainer" containerID="e8a2d8dcad1c5723e706475af6eafee87f8d680979c575f13b0976abadb872f0"
Oct 30 01:19:04.075310 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.075282    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"]
Oct 30 01:19:04.075451 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.075435    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.080989 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:19:04.079623    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a2d8dcad1c5723e706475af6eafee87f8d680979c575f13b0976abadb872f0\": container with ID starting with e8a2d8dcad1c5723e706475af6eafee87f8d680979c575f13b0976abadb872f0 not found: ID does not exist" containerID="e8a2d8dcad1c5723e706475af6eafee87f8d680979c575f13b0976abadb872f0"
Oct 30 01:19:04.080989 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.079668    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a2d8dcad1c5723e706475af6eafee87f8d680979c575f13b0976abadb872f0"} err="failed to get container status \"e8a2d8dcad1c5723e706475af6eafee87f8d680979c575f13b0976abadb872f0\": rpc error: code = NotFound desc = could not find container \"e8a2d8dcad1c5723e706475af6eafee87f8d680979c575f13b0976abadb872f0\": container with ID starting with e8a2d8dcad1c5723e706475af6eafee87f8d680979c575f13b0976abadb872f0 not found: ID does not exist"
Oct 30 01:19:04.173103 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.173022    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0bbb319-2301-41ef-879f-29ea51d8699e-istiod-ca-cert\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.173103 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.173070    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-token\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.173289 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.173229    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-workload-socket\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.173289 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.173261    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-podinfo\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.173376 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.173285    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-data\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.173435 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.173412    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-credential-socket\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.173498 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.173438    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r7bt\" (UniqueName: \"kubernetes.io/projected/f0bbb319-2301-41ef-879f-29ea51d8699e-kube-api-access-7r7bt\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.173597 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.173492    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-envoy\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.173668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.173609    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-workload-certs\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.274874 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.274843    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0bbb319-2301-41ef-879f-29ea51d8699e-istiod-ca-cert\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.275224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.274893    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-token\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.275224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.274987    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-workload-socket\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.275224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.275009    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-podinfo\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.275224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.275031    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-data\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.275224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.275109    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-credential-socket\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.275224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.275152    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r7bt\" (UniqueName: \"kubernetes.io/projected/f0bbb319-2301-41ef-879f-29ea51d8699e-kube-api-access-7r7bt\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.275586 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.275230    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-envoy\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.275586 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.275336    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-workload-socket\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.275586 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.275473    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-workload-certs\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.275586 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.275547    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-data\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.275586 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.275576    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0bbb319-2301-41ef-879f-29ea51d8699e-istiod-ca-cert\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.276483 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.275771    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-workload-certs\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.276483 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.275816    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-credential-socket\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.277565 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.277541    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-envoy\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.277761 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.277741    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-podinfo\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.298089 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.298068    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-token\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.298218 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.298201    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r7bt\" (UniqueName: \"kubernetes.io/projected/f0bbb319-2301-41ef-879f-29ea51d8699e-kube-api-access-7r7bt\") pod \"e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") " pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.400809 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.400780    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:04.727844 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:19:04.727818    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bbb319_2301_41ef_879f_29ea51d8699e.slice/crio-c30d654a412b5da137f7f2522113e1af04ac227f5dd85984868150827a2ba7e0 WatchSource:0}: Error finding container c30d654a412b5da137f7f2522113e1af04ac227f5dd85984868150827a2ba7e0: Status 404 returned error can't find the container with id c30d654a412b5da137f7f2522113e1af04ac227f5dd85984868150827a2ba7e0
Oct 30 01:19:04.728229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.728207    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"]
Oct 30 01:19:04.730604 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.730573    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:19:04.730664 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.730642    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:19:04.730722 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.730691    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:19:04.869350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.869312    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss" event={"ID":"f0bbb319-2301-41ef-879f-29ea51d8699e","Type":"ContainerStarted","Data":"27a204116bf3ae3983e66ccc6968a3b568297d72a9a11ce458ebf65836f5621a"}
Oct 30 01:19:04.869555 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.869377    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss" event={"ID":"f0bbb319-2301-41ef-879f-29ea51d8699e","Type":"ContainerStarted","Data":"c30d654a412b5da137f7f2522113e1af04ac227f5dd85984868150827a2ba7e0"}
Oct 30 01:19:04.870582 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.870560    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"
Oct 30 01:19:04.916179 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.916130    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss" podStartSLOduration=59.916115314 podStartE2EDuration="59.916115314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:04.913661822 +0000 UTC m=+2638.770569590" watchObservedRunningTime="2026-10-30 01:19:04.916115314 +0000 UTC m=+2638.773023132"
Oct 30 01:19:04.955917 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.955882    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"]
Oct 30 01:19:04.959664 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:04.959639    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-320045-745dbd46d-7zs7w"]
Oct 30 01:19:05.401344 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:05.401305    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:05.402758 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:05.402734    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Oct 30 01:19:05.402887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:05.402781    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss" podUID="f0bbb319-2301-41ef-879f-29ea51d8699e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Oct 30 01:19:06.290018 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:06.289969    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b07e253-0493-423b-a506-eaebd9952e34" path="/var/lib/kubelet/pods/5b07e253-0493-423b-a506-eaebd9952e34/volumes"
Oct 30 01:19:06.401655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:06.401621    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Oct 30 01:19:06.402064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:06.401680    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss" podUID="f0bbb319-2301-41ef-879f-29ea51d8699e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Oct 30 01:19:07.401174 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:07.401137    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Oct 30 01:19:07.401367 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:07.401192    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss" podUID="f0bbb319-2301-41ef-879f-29ea51d8699e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Oct 30 01:19:08.402037 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:08.402002    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Oct 30 01:19:08.402444 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:08.402059    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss" podUID="f0bbb319-2301-41ef-879f-29ea51d8699e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Oct 30 01:19:09.401410 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:09.401373    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Oct 30 01:19:09.401609 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:09.401421    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss" podUID="f0bbb319-2301-41ef-879f-29ea51d8699e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Oct 30 01:19:10.401456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:10.401421    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Oct 30 01:19:10.401935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:10.401473    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss" podUID="f0bbb319-2301-41ef-879f-29ea51d8699e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Oct 30 01:19:11.342772 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:11.328844    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"]
Oct 30 01:19:11.342772 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:11.330126    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l" podUID="49482cc2-07e6-455e-952e-bd42810e8d7b" containerName="istio-proxy" containerID="cri-o://bada115a463ac0a8ce34cd05d86de2cbfae7bd94d16fece22f9e8874a3d98c8c" gracePeriod=30
Oct 30 01:19:11.359444 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:11.359410    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"]
Oct 30 01:19:11.360537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:11.359991    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7" podUID="cc28ff31-b61a-4d8e-9ff3-361345eb9c06" containerName="istio-proxy" containerID="cri-o://afda7568d68e390d0b0a7b46b543e5a2ad087b6448e740fe8221162f1de57775" gracePeriod=30
Oct 30 01:19:11.377984 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:11.377951    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"]
Oct 30 01:19:11.387028 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:11.386993    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"]
Oct 30 01:19:11.404395 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:11.403625    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs" podUID="a528dfda-e8fb-4a8e-abe7-4d870b94a64a" containerName="istio-proxy" containerID="cri-o://865c196c81a38de3defa6e460c36ec447807e39d28e0b7c2ce9155138c26e0f0" gracePeriod=30
Oct 30 01:19:11.405677 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:11.405046    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg" podUID="4f42de42-9772-41e0-9cdb-8a9b212106ee" containerName="istio-proxy" containerID="cri-o://73e0372ba6604553dadd0e370509ed9ee7cc6dc08a04333fac52e458ca36a7d0" gracePeriod=30
Oct 30 01:19:11.420886 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:11.416768    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:11.420886 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:11.417689    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:11.439779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:11.431798    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:19:16.652737 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:19:16.651974    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:19:16.752988 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.752002    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:19:16.838919 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.838882    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"]
Oct 30 01:19:16.840415 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.840387    6099 cpu_manager.go:414] "containerMap: removing container" podUID="cc28ff31-b61a-4d8e-9ff3-361345eb9c06" containerName="istio-proxy"
Oct 30 01:19:16.840415 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.840413    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc28ff31-b61a-4d8e-9ff3-361345eb9c06" containerName="istio-proxy"
Oct 30 01:19:16.840626 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.840435    6099 cpu_manager.go:414] "containerMap: removing container" podUID="cc28ff31-b61a-4d8e-9ff3-361345eb9c06" containerName="istio-proxy"
Oct 30 01:19:16.840626 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.840444    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc28ff31-b61a-4d8e-9ff3-361345eb9c06" containerName="istio-proxy"
Oct 30 01:19:16.841149 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.841129    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc28ff31-b61a-4d8e-9ff3-361345eb9c06" containerName="istio-proxy"
Oct 30 01:19:16.841315 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.841301    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc28ff31-b61a-4d8e-9ff3-361345eb9c06" containerName="istio-proxy"
Oct 30 01:19:16.845275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.845218    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:19:16.850380 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.849482    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-kube-api-access-cp8wp\" (UniqueName: \"kubernetes.io/projected/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-kube-api-access-cp8wp\") pod \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") "
Oct 30 01:19:16.850380 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.849529    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-workload-certs\") pod \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") "
Oct 30 01:19:16.850380 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.849626    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-token\" (UniqueName: \"kubernetes.io/projected/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-token\") pod \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") "
Oct 30 01:19:16.850380 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.849756    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-workload-socket\") pod \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") "
Oct 30 01:19:16.850380 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.849845    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-podinfo\") pod \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") "
Oct 30 01:19:16.850380 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.849885    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-credential-socket\") pod \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") "
Oct 30 01:19:16.850380 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.849985    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-data\") pod \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") "
Oct 30 01:19:16.850380 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.850022    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-envoy\") pod \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") "
Oct 30 01:19:16.850380 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.850073    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istiod-ca-cert\") pod \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\" (UID: \"cc28ff31-b61a-4d8e-9ff3-361345eb9c06\") "
Oct 30 01:19:16.855953 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.855920    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istiod-ca-cert" pod "cc28ff31-b61a-4d8e-9ff3-361345eb9c06" (UID: "cc28ff31-b61a-4d8e-9ff3-361345eb9c06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:16.858426 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.858391    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-workload-certs" pod "cc28ff31-b61a-4d8e-9ff3-361345eb9c06" (UID: "cc28ff31-b61a-4d8e-9ff3-361345eb9c06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:16.860134 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.859100    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-workload-socket" pod "cc28ff31-b61a-4d8e-9ff3-361345eb9c06" (UID: "cc28ff31-b61a-4d8e-9ff3-361345eb9c06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:16.860134 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.859126    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-credential-socket" pod "cc28ff31-b61a-4d8e-9ff3-361345eb9c06" (UID: "cc28ff31-b61a-4d8e-9ff3-361345eb9c06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:16.860134 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.859142    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-data" pod "cc28ff31-b61a-4d8e-9ff3-361345eb9c06" (UID: "cc28ff31-b61a-4d8e-9ff3-361345eb9c06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:16.863171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.863046    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-envoy" pod "cc28ff31-b61a-4d8e-9ff3-361345eb9c06" (UID: "cc28ff31-b61a-4d8e-9ff3-361345eb9c06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:16.863291 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.863146    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-podinfo" pod "cc28ff31-b61a-4d8e-9ff3-361345eb9c06" (UID: "cc28ff31-b61a-4d8e-9ff3-361345eb9c06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:16.863548 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.863185    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-token" pod "cc28ff31-b61a-4d8e-9ff3-361345eb9c06" (UID: "cc28ff31-b61a-4d8e-9ff3-361345eb9c06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:16.863675 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.863653    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-kube-api-access-cp8wp" pod "cc28ff31-b61a-4d8e-9ff3-361345eb9c06" (UID: "cc28ff31-b61a-4d8e-9ff3-361345eb9c06"). InnerVolumeSpecName "kube-api-access-cp8wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:16.867413 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.867389    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.867515 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.867450    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.867682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.867469    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.867757 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.867694    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.868223 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.868200    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.868302 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.868230    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.868302 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.868261    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp8wp\" (UniqueName: \"kubernetes.io/projected/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-kube-api-access-cp8wp\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.868509 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.868490    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.868567 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.868524    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc28ff31-b61a-4d8e-9ff3-361345eb9c06-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.885562 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.885537    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:19:16.885884 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.885861    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"]
Oct 30 01:19:16.886000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.885957    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"
Oct 30 01:19:16.887858 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.887836    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:19:16.901376 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.901352    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-38b47d-serving-cert\""
Oct 30 01:19:16.916062 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.916029    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m"]
Oct 30 01:19:16.917252 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.917226    6099 cpu_manager.go:414] "containerMap: removing container" podUID="a528dfda-e8fb-4a8e-abe7-4d870b94a64a" containerName="istio-proxy"
Oct 30 01:19:16.919379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.917254    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a528dfda-e8fb-4a8e-abe7-4d870b94a64a" containerName="istio-proxy"
Oct 30 01:19:16.919379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.917273    6099 cpu_manager.go:414] "containerMap: removing container" podUID="49482cc2-07e6-455e-952e-bd42810e8d7b" containerName="istio-proxy"
Oct 30 01:19:16.919379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.917281    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49482cc2-07e6-455e-952e-bd42810e8d7b" containerName="istio-proxy"
Oct 30 01:19:16.919379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.917335    6099 cpu_manager.go:414] "containerMap: removing container" podUID="4f42de42-9772-41e0-9cdb-8a9b212106ee" containerName="istio-proxy"
Oct 30 01:19:16.919379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.917341    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f42de42-9772-41e0-9cdb-8a9b212106ee" containerName="istio-proxy"
Oct 30 01:19:16.919379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.917470    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f42de42-9772-41e0-9cdb-8a9b212106ee" containerName="istio-proxy"
Oct 30 01:19:16.919379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.917481    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a528dfda-e8fb-4a8e-abe7-4d870b94a64a" containerName="istio-proxy"
Oct 30 01:19:16.919379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.917489    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="49482cc2-07e6-455e-952e-bd42810e8d7b" containerName="istio-proxy"
Oct 30 01:19:16.922260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.922241    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m"
Oct 30 01:19:16.940888 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.940859    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m"]
Oct 30 01:19:16.969177 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.969141    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-credential-socket\") pod \"49482cc2-07e6-455e-952e-bd42810e8d7b\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") "
Oct 30 01:19:16.969387 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.969195    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-envoy\") pod \"49482cc2-07e6-455e-952e-bd42810e8d7b\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") "
Oct 30 01:19:16.969514 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.969455    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-data\") pod \"49482cc2-07e6-455e-952e-bd42810e8d7b\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") "
Oct 30 01:19:16.969962 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.969452    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-credential-socket" pod "49482cc2-07e6-455e-952e-bd42810e8d7b" (UID: "49482cc2-07e6-455e-952e-bd42810e8d7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:16.970077 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.969637    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-data" pod "49482cc2-07e6-455e-952e-bd42810e8d7b" (UID: "49482cc2-07e6-455e-952e-bd42810e8d7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:16.970077 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.969991    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-workload-socket\") pod \"49482cc2-07e6-455e-952e-bd42810e8d7b\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") "
Oct 30 01:19:16.970197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.970142    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-token\" (UniqueName: \"kubernetes.io/projected/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-token\") pod \"49482cc2-07e6-455e-952e-bd42810e8d7b\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") "
Oct 30 01:19:16.970254 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.970218    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-workload-socket" pod "49482cc2-07e6-455e-952e-bd42810e8d7b" (UID: "49482cc2-07e6-455e-952e-bd42810e8d7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:16.970572 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.970550    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-workload-certs\") pod \"49482cc2-07e6-455e-952e-bd42810e8d7b\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") "
Oct 30 01:19:16.970860 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.970831    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-workload-certs" pod "49482cc2-07e6-455e-952e-bd42810e8d7b" (UID: "49482cc2-07e6-455e-952e-bd42810e8d7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:16.971065 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.970843    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49482cc2-07e6-455e-952e-bd42810e8d7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49482cc2-07e6-455e-952e-bd42810e8d7b-istiod-ca-cert\") pod \"49482cc2-07e6-455e-952e-bd42810e8d7b\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") "
Oct 30 01:19:16.971065 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.970951    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-podinfo\") pod \"49482cc2-07e6-455e-952e-bd42810e8d7b\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") "
Oct 30 01:19:16.971273 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.971064    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49482cc2-07e6-455e-952e-bd42810e8d7b-kube-api-access-2gfh5\" (UniqueName: \"kubernetes.io/projected/49482cc2-07e6-455e-952e-bd42810e8d7b-kube-api-access-2gfh5\") pod \"49482cc2-07e6-455e-952e-bd42810e8d7b\" (UID: \"49482cc2-07e6-455e-952e-bd42810e8d7b\") "
Oct 30 01:19:16.971273 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.971150    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49482cc2-07e6-455e-952e-bd42810e8d7b-istiod-ca-cert" pod "49482cc2-07e6-455e-952e-bd42810e8d7b" (UID: "49482cc2-07e6-455e-952e-bd42810e8d7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:16.971860 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.971835    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-envoy" pod "49482cc2-07e6-455e-952e-bd42810e8d7b" (UID: "49482cc2-07e6-455e-952e-bd42810e8d7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:16.971988 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.971968    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.972063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.971994    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49482cc2-07e6-455e-952e-bd42810e8d7b-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.972063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.972008    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.972063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.972021    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.972063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.972036    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.972291 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.972064    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49482cc2-07e6-455e-952e-bd42810e8d7b-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.972517 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.972485    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-token" pod "49482cc2-07e6-455e-952e-bd42810e8d7b" (UID: "49482cc2-07e6-455e-952e-bd42810e8d7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:16.973664 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.973639    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-podinfo" pod "49482cc2-07e6-455e-952e-bd42810e8d7b" (UID: "49482cc2-07e6-455e-952e-bd42810e8d7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:16.973917 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:16.973896    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49482cc2-07e6-455e-952e-bd42810e8d7b-kube-api-access-2gfh5" pod "49482cc2-07e6-455e-952e-bd42810e8d7b" (UID: "49482cc2-07e6-455e-952e-bd42810e8d7b"). InnerVolumeSpecName "kube-api-access-2gfh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:17.007106 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.007076    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b"]
Oct 30 01:19:17.017073 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.017048    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b"
Oct 30 01:19:17.025091 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.025060    6099 generic.go:356] "Generic (PLEG): container finished" podID="cc28ff31-b61a-4d8e-9ff3-361345eb9c06" containerID="afda7568d68e390d0b0a7b46b543e5a2ad087b6448e740fe8221162f1de57775" exitCode=0
Oct 30 01:19:17.025216 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.025181    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7" event={"ID":"cc28ff31-b61a-4d8e-9ff3-361345eb9c06","Type":"ContainerDied","Data":"afda7568d68e390d0b0a7b46b543e5a2ad087b6448e740fe8221162f1de57775"}
Oct 30 01:19:17.025260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.025224    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7" event={"ID":"cc28ff31-b61a-4d8e-9ff3-361345eb9c06","Type":"ContainerDied","Data":"9463b6c0af14f2cd3fc7fe52febb8264edc21f80d36e38d68fdb3069a9cf22d8"}
Oct 30 01:19:17.025260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.025254    6099 scope.go:122] "RemoveContainer" containerID="afda7568d68e390d0b0a7b46b543e5a2ad087b6448e740fe8221162f1de57775"
Oct 30 01:19:17.025440 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.025424    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"
Oct 30 01:19:17.039359 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.039331    6099 generic.go:356] "Generic (PLEG): container finished" podID="a528dfda-e8fb-4a8e-abe7-4d870b94a64a" containerID="865c196c81a38de3defa6e460c36ec447807e39d28e0b7c2ce9155138c26e0f0" exitCode=0
Oct 30 01:19:17.039513 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.039491    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs" event={"ID":"a528dfda-e8fb-4a8e-abe7-4d870b94a64a","Type":"ContainerDied","Data":"865c196c81a38de3defa6e460c36ec447807e39d28e0b7c2ce9155138c26e0f0"}
Oct 30 01:19:17.039600 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.039542    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs" event={"ID":"a528dfda-e8fb-4a8e-abe7-4d870b94a64a","Type":"ContainerDied","Data":"854de2b4a9143c279e624e69b4a20ff5ecedf3d7fd689e7761a26c84fbc25a80"}
Oct 30 01:19:17.039663 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.039628    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"
Oct 30 01:19:17.043875 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.041905    6099 generic.go:356] "Generic (PLEG): container finished" podID="4f42de42-9772-41e0-9cdb-8a9b212106ee" containerID="73e0372ba6604553dadd0e370509ed9ee7cc6dc08a04333fac52e458ca36a7d0" exitCode=0
Oct 30 01:19:17.043875 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.041986    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg" event={"ID":"4f42de42-9772-41e0-9cdb-8a9b212106ee","Type":"ContainerDied","Data":"73e0372ba6604553dadd0e370509ed9ee7cc6dc08a04333fac52e458ca36a7d0"}
Oct 30 01:19:17.043875 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.042026    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg" event={"ID":"4f42de42-9772-41e0-9cdb-8a9b212106ee","Type":"ContainerDied","Data":"7aff112a334002f3aea1226921fbf581be492088b2c9f4681989e7df408209d4"}
Oct 30 01:19:17.043875 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.042100    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"
Oct 30 01:19:17.043875 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.043687    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-abf8ae\""
Oct 30 01:19:17.044756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.044735    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b"]
Oct 30 01:19:17.045757 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.045699    6099 generic.go:356] "Generic (PLEG): container finished" podID="49482cc2-07e6-455e-952e-bd42810e8d7b" containerID="bada115a463ac0a8ce34cd05d86de2cbfae7bd94d16fece22f9e8874a3d98c8c" exitCode=0
Oct 30 01:19:17.045855 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.045836    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l" event={"ID":"49482cc2-07e6-455e-952e-bd42810e8d7b","Type":"ContainerDied","Data":"bada115a463ac0a8ce34cd05d86de2cbfae7bd94d16fece22f9e8874a3d98c8c"}
Oct 30 01:19:17.045917 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.045875    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l" event={"ID":"49482cc2-07e6-455e-952e-bd42810e8d7b","Type":"ContainerDied","Data":"8994b89f6ae26e1b7eca222933af3e7e89d73b9fce4235e9b30a11f9d70c3a92"}
Oct 30 01:19:17.046016 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.046001    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"
Oct 30 01:19:17.068803 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.068783    6099 scope.go:122] "RemoveContainer" containerID="023512b0f3b8ca26813352bab4f7f74e5b2c877a51d9d733976ef77ae1fd5f58"
Oct 30 01:19:17.073340 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.073319    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f42de42-9772-41e0-9cdb-8a9b212106ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f42de42-9772-41e0-9cdb-8a9b212106ee-istiod-ca-cert\") pod \"4f42de42-9772-41e0-9cdb-8a9b212106ee\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") "
Oct 30 01:19:17.073605 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.073587    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istiod-ca-cert\") pod \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") "
Oct 30 01:19:17.073751 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.073736    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-data\") pod \"4f42de42-9772-41e0-9cdb-8a9b212106ee\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") "
Oct 30 01:19:17.073908 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.073893    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-podinfo\") pod \"4f42de42-9772-41e0-9cdb-8a9b212106ee\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") "
Oct 30 01:19:17.074026 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.074011    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-data\") pod \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") "
Oct 30 01:19:17.074157 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.074143    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-credential-socket\") pod \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") "
Oct 30 01:19:17.074330 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.074314    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-envoy\") pod \"4f42de42-9772-41e0-9cdb-8a9b212106ee\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") "
Oct 30 01:19:17.074484 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.074445    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-data" pod "4f42de42-9772-41e0-9cdb-8a9b212106ee" (UID: "4f42de42-9772-41e0-9cdb-8a9b212106ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:17.074575 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.074473    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-workload-certs\") pod \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") "
Oct 30 01:19:17.074834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.074814    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-kube-api-access-mk9ss\" (UniqueName: \"kubernetes.io/projected/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-kube-api-access-mk9ss\") pod \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") "
Oct 30 01:19:17.074988 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.074972    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-workload-socket\") pod \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") "
Oct 30 01:19:17.075145 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.075127    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-workload-socket\") pod \"4f42de42-9772-41e0-9cdb-8a9b212106ee\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") "
Oct 30 01:19:17.075291 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.075273    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f42de42-9772-41e0-9cdb-8a9b212106ee-kube-api-access-xnbjk\" (UniqueName: \"kubernetes.io/projected/4f42de42-9772-41e0-9cdb-8a9b212106ee-kube-api-access-xnbjk\") pod \"4f42de42-9772-41e0-9cdb-8a9b212106ee\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") "
Oct 30 01:19:17.075475 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.075457    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-credential-socket\") pod \"4f42de42-9772-41e0-9cdb-8a9b212106ee\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") "
Oct 30 01:19:17.075679 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.075650    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-token\" (UniqueName: \"kubernetes.io/projected/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-token\") pod \"4f42de42-9772-41e0-9cdb-8a9b212106ee\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") "
Oct 30 01:19:17.075874 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.075778    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-token\" (UniqueName: \"kubernetes.io/projected/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-token\") pod \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") "
Oct 30 01:19:17.075874 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.074837    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-credential-socket" pod "a528dfda-e8fb-4a8e-abe7-4d870b94a64a" (UID: "a528dfda-e8fb-4a8e-abe7-4d870b94a64a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:17.075874 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.075875    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istiod-ca-cert" pod "a528dfda-e8fb-4a8e-abe7-4d870b94a64a" (UID: "a528dfda-e8fb-4a8e-abe7-4d870b94a64a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:17.076616 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.075028    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-data" pod "a528dfda-e8fb-4a8e-abe7-4d870b94a64a" (UID: "a528dfda-e8fb-4a8e-abe7-4d870b94a64a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:17.076616 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.075921    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-podinfo\") pod \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") "
Oct 30 01:19:17.076616 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.075956    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-workload-certs\") pod \"4f42de42-9772-41e0-9cdb-8a9b212106ee\" (UID: \"4f42de42-9772-41e0-9cdb-8a9b212106ee\") "
Oct 30 01:19:17.076616 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.075997    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-envoy\") pod \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\" (UID: \"a528dfda-e8fb-4a8e-abe7-4d870b94a64a\") "
Oct 30 01:19:17.076616 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.076501    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1272be00-9aa6-4cb1-9d1e-74cfe0164569-metrics-certs\") pod \"maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q\" (UID: \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"
Oct 30 01:19:17.076616 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.076555    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txlkg\" (UniqueName: \"kubernetes.io/projected/1272be00-9aa6-4cb1-9d1e-74cfe0164569-kube-api-access-txlkg\") pod \"maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q\" (UID: \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"
Oct 30 01:19:17.076963 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.076676    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1272be00-9aa6-4cb1-9d1e-74cfe0164569-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q\" (UID: \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"
Oct 30 01:19:17.076963 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.076873    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1272be00-9aa6-4cb1-9d1e-74cfe0164569-maas-api-tls\") pod \"maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q\" (UID: \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"
Oct 30 01:19:17.077120 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.077052    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl9fk\" (UniqueName: \"kubernetes.io/projected/261fed99-0c9e-42cb-ad91-540d2b125576-kube-api-access-gl9fk\") pod \"payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b\" (UID: \"261fed99-0c9e-42cb-ad91-540d2b125576\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b"
Oct 30 01:19:17.077120 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.077115    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9k7f\" (UniqueName: \"kubernetes.io/projected/57a0acc3-77a4-4ad2-978e-08280840d903-kube-api-access-x9k7f\") pod \"payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m\" (UID: \"57a0acc3-77a4-4ad2-978e-08280840d903\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m"
Oct 30 01:19:17.077244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.077192    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a0acc3-77a4-4ad2-978e-08280840d903-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m\" (UID: \"57a0acc3-77a4-4ad2-978e-08280840d903\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m"
Oct 30 01:19:17.077244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.077226    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261fed99-0c9e-42cb-ad91-540d2b125576-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b\" (UID: \"261fed99-0c9e-42cb-ad91-540d2b125576\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b"
Oct 30 01:19:17.077350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.077323    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.077350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.077338    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49482cc2-07e6-455e-952e-bd42810e8d7b-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.077465 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.077354    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gfh5\" (UniqueName: \"kubernetes.io/projected/49482cc2-07e6-455e-952e-bd42810e8d7b-kube-api-access-2gfh5\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.077465 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.077370    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.077465 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.077385    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.077465 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.077399    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.077465 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.077414    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.077465 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.075529    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f42de42-9772-41e0-9cdb-8a9b212106ee-istiod-ca-cert" pod "4f42de42-9772-41e0-9cdb-8a9b212106ee" (UID: "4f42de42-9772-41e0-9cdb-8a9b212106ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:17.077465 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.075818    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-workload-socket" pod "a528dfda-e8fb-4a8e-abe7-4d870b94a64a" (UID: "a528dfda-e8fb-4a8e-abe7-4d870b94a64a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:17.077985 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.076117    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-workload-certs" pod "a528dfda-e8fb-4a8e-abe7-4d870b94a64a" (UID: "a528dfda-e8fb-4a8e-abe7-4d870b94a64a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:17.077985 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.076229    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-credential-socket" pod "4f42de42-9772-41e0-9cdb-8a9b212106ee" (UID: "4f42de42-9772-41e0-9cdb-8a9b212106ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:17.077985 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.076537    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-workload-socket" pod "4f42de42-9772-41e0-9cdb-8a9b212106ee" (UID: "4f42de42-9772-41e0-9cdb-8a9b212106ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:17.077985 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.077692    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-workload-certs" pod "4f42de42-9772-41e0-9cdb-8a9b212106ee" (UID: "4f42de42-9772-41e0-9cdb-8a9b212106ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:17.094006 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.093906    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-token" pod "4f42de42-9772-41e0-9cdb-8a9b212106ee" (UID: "4f42de42-9772-41e0-9cdb-8a9b212106ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:17.094006 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.093960    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f42de42-9772-41e0-9cdb-8a9b212106ee-kube-api-access-xnbjk" pod "4f42de42-9772-41e0-9cdb-8a9b212106ee" (UID: "4f42de42-9772-41e0-9cdb-8a9b212106ee"). InnerVolumeSpecName "kube-api-access-xnbjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:17.094006 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.093986    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-token" pod "a528dfda-e8fb-4a8e-abe7-4d870b94a64a" (UID: "a528dfda-e8fb-4a8e-abe7-4d870b94a64a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:17.095842 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.094823    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-envoy" pod "4f42de42-9772-41e0-9cdb-8a9b212106ee" (UID: "4f42de42-9772-41e0-9cdb-8a9b212106ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:17.095842 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.094877    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-kube-api-access-mk9ss" pod "a528dfda-e8fb-4a8e-abe7-4d870b94a64a" (UID: "a528dfda-e8fb-4a8e-abe7-4d870b94a64a"). InnerVolumeSpecName "kube-api-access-mk9ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:17.095842 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.094932    6099 scope.go:122] "RemoveContainer" containerID="afda7568d68e390d0b0a7b46b543e5a2ad087b6448e740fe8221162f1de57775"
Oct 30 01:19:17.095842 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.094969    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-podinfo" pod "4f42de42-9772-41e0-9cdb-8a9b212106ee" (UID: "4f42de42-9772-41e0-9cdb-8a9b212106ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:17.095842 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:19:17.095321    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afda7568d68e390d0b0a7b46b543e5a2ad087b6448e740fe8221162f1de57775\": container with ID starting with afda7568d68e390d0b0a7b46b543e5a2ad087b6448e740fe8221162f1de57775 not found: ID does not exist" containerID="afda7568d68e390d0b0a7b46b543e5a2ad087b6448e740fe8221162f1de57775"
Oct 30 01:19:17.095842 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.095359    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afda7568d68e390d0b0a7b46b543e5a2ad087b6448e740fe8221162f1de57775"} err="failed to get container status \"afda7568d68e390d0b0a7b46b543e5a2ad087b6448e740fe8221162f1de57775\": rpc error: code = NotFound desc = could not find container \"afda7568d68e390d0b0a7b46b543e5a2ad087b6448e740fe8221162f1de57775\": container with ID starting with afda7568d68e390d0b0a7b46b543e5a2ad087b6448e740fe8221162f1de57775 not found: ID does not exist"
Oct 30 01:19:17.095842 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.095387    6099 scope.go:122] "RemoveContainer" containerID="023512b0f3b8ca26813352bab4f7f74e5b2c877a51d9d733976ef77ae1fd5f58"
Oct 30 01:19:17.095842 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:19:17.095723    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"023512b0f3b8ca26813352bab4f7f74e5b2c877a51d9d733976ef77ae1fd5f58\": container with ID starting with 023512b0f3b8ca26813352bab4f7f74e5b2c877a51d9d733976ef77ae1fd5f58 not found: ID does not exist" containerID="023512b0f3b8ca26813352bab4f7f74e5b2c877a51d9d733976ef77ae1fd5f58"
Oct 30 01:19:17.095842 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.095752    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"023512b0f3b8ca26813352bab4f7f74e5b2c877a51d9d733976ef77ae1fd5f58"} err="failed to get container status \"023512b0f3b8ca26813352bab4f7f74e5b2c877a51d9d733976ef77ae1fd5f58\": rpc error: code = NotFound desc = could not find container \"023512b0f3b8ca26813352bab4f7f74e5b2c877a51d9d733976ef77ae1fd5f58\": container with ID starting with 023512b0f3b8ca26813352bab4f7f74e5b2c877a51d9d733976ef77ae1fd5f58 not found: ID does not exist"
Oct 30 01:19:17.095842 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.095772    6099 scope.go:122] "RemoveContainer" containerID="865c196c81a38de3defa6e460c36ec447807e39d28e0b7c2ce9155138c26e0f0"
Oct 30 01:19:17.097158 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.097132    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-envoy" pod "a528dfda-e8fb-4a8e-abe7-4d870b94a64a" (UID: "a528dfda-e8fb-4a8e-abe7-4d870b94a64a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:17.097856 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.097831    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-podinfo" pod "a528dfda-e8fb-4a8e-abe7-4d870b94a64a" (UID: "a528dfda-e8fb-4a8e-abe7-4d870b94a64a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:17.104360 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.104308    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-abf8ae-dockercfg-dhssg\""
Oct 30 01:19:17.112058 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.111690    6099 scope.go:122] "RemoveContainer" containerID="865c196c81a38de3defa6e460c36ec447807e39d28e0b7c2ce9155138c26e0f0"
Oct 30 01:19:17.121870 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:19:17.121841    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865c196c81a38de3defa6e460c36ec447807e39d28e0b7c2ce9155138c26e0f0\": container with ID starting with 865c196c81a38de3defa6e460c36ec447807e39d28e0b7c2ce9155138c26e0f0 not found: ID does not exist" containerID="865c196c81a38de3defa6e460c36ec447807e39d28e0b7c2ce9155138c26e0f0"
Oct 30 01:19:17.121978 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.121880    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865c196c81a38de3defa6e460c36ec447807e39d28e0b7c2ce9155138c26e0f0"} err="failed to get container status \"865c196c81a38de3defa6e460c36ec447807e39d28e0b7c2ce9155138c26e0f0\": rpc error: code = NotFound desc = could not find container \"865c196c81a38de3defa6e460c36ec447807e39d28e0b7c2ce9155138c26e0f0\": container with ID starting with 865c196c81a38de3defa6e460c36ec447807e39d28e0b7c2ce9155138c26e0f0 not found: ID does not exist"
Oct 30 01:19:17.121978 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.121907    6099 scope.go:122] "RemoveContainer" containerID="73e0372ba6604553dadd0e370509ed9ee7cc6dc08a04333fac52e458ca36a7d0"
Oct 30 01:19:17.128511 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:19:17.128484    6099 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49482cc2_07e6_455e_952e_bd42810e8d7b.slice/crio-8994b89f6ae26e1b7eca222933af3e7e89d73b9fce4235e9b30a11f9d70c3a92\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49482cc2_07e6_455e_952e_bd42810e8d7b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc28ff31_b61a_4d8e_9ff3_361345eb9c06.slice/crio-9463b6c0af14f2cd3fc7fe52febb8264edc21f80d36e38d68fdb3069a9cf22d8\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc28ff31_b61a_4d8e_9ff3_361345eb9c06.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 01:19:17.136511 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.136489    6099 scope.go:122] "RemoveContainer" containerID="73e0372ba6604553dadd0e370509ed9ee7cc6dc08a04333fac52e458ca36a7d0"
Oct 30 01:19:17.136868 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:19:17.136846    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e0372ba6604553dadd0e370509ed9ee7cc6dc08a04333fac52e458ca36a7d0\": container with ID starting with 73e0372ba6604553dadd0e370509ed9ee7cc6dc08a04333fac52e458ca36a7d0 not found: ID does not exist" containerID="73e0372ba6604553dadd0e370509ed9ee7cc6dc08a04333fac52e458ca36a7d0"
Oct 30 01:19:17.136952 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.136876    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e0372ba6604553dadd0e370509ed9ee7cc6dc08a04333fac52e458ca36a7d0"} err="failed to get container status \"73e0372ba6604553dadd0e370509ed9ee7cc6dc08a04333fac52e458ca36a7d0\": rpc error: code = NotFound desc = could not find container \"73e0372ba6604553dadd0e370509ed9ee7cc6dc08a04333fac52e458ca36a7d0\": container with ID starting with 73e0372ba6604553dadd0e370509ed9ee7cc6dc08a04333fac52e458ca36a7d0 not found: ID does not exist"
Oct 30 01:19:17.136952 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.136892    6099 scope.go:122] "RemoveContainer" containerID="bada115a463ac0a8ce34cd05d86de2cbfae7bd94d16fece22f9e8874a3d98c8c"
Oct 30 01:19:17.148843 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.148827    6099 scope.go:122] "RemoveContainer" containerID="bada115a463ac0a8ce34cd05d86de2cbfae7bd94d16fece22f9e8874a3d98c8c"
Oct 30 01:19:17.149175 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:19:17.149142    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bada115a463ac0a8ce34cd05d86de2cbfae7bd94d16fece22f9e8874a3d98c8c\": container with ID starting with bada115a463ac0a8ce34cd05d86de2cbfae7bd94d16fece22f9e8874a3d98c8c not found: ID does not exist" containerID="bada115a463ac0a8ce34cd05d86de2cbfae7bd94d16fece22f9e8874a3d98c8c"
Oct 30 01:19:17.149297 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.149182    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bada115a463ac0a8ce34cd05d86de2cbfae7bd94d16fece22f9e8874a3d98c8c"} err="failed to get container status \"bada115a463ac0a8ce34cd05d86de2cbfae7bd94d16fece22f9e8874a3d98c8c\": rpc error: code = NotFound desc = could not find container \"bada115a463ac0a8ce34cd05d86de2cbfae7bd94d16fece22f9e8874a3d98c8c\": container with ID starting with bada115a463ac0a8ce34cd05d86de2cbfae7bd94d16fece22f9e8874a3d98c8c not found: ID does not exist"
Oct 30 01:19:17.179199 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.179176    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1272be00-9aa6-4cb1-9d1e-74cfe0164569-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q\" (UID: \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"
Oct 30 01:19:17.179325 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.179306    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1272be00-9aa6-4cb1-9d1e-74cfe0164569-maas-api-tls\") pod \"maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q\" (UID: \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"
Oct 30 01:19:17.179457 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.179439    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl9fk\" (UniqueName: \"kubernetes.io/projected/261fed99-0c9e-42cb-ad91-540d2b125576-kube-api-access-gl9fk\") pod \"payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b\" (UID: \"261fed99-0c9e-42cb-ad91-540d2b125576\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b"
Oct 30 01:19:17.179508 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.179492    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9k7f\" (UniqueName: \"kubernetes.io/projected/57a0acc3-77a4-4ad2-978e-08280840d903-kube-api-access-x9k7f\") pod \"payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m\" (UID: \"57a0acc3-77a4-4ad2-978e-08280840d903\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m"
Oct 30 01:19:17.179568 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.179553    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a0acc3-77a4-4ad2-978e-08280840d903-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m\" (UID: \"57a0acc3-77a4-4ad2-978e-08280840d903\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m"
Oct 30 01:19:17.179630 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.179586    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261fed99-0c9e-42cb-ad91-540d2b125576-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b\" (UID: \"261fed99-0c9e-42cb-ad91-540d2b125576\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b"
Oct 30 01:19:17.179778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.179756    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1272be00-9aa6-4cb1-9d1e-74cfe0164569-metrics-certs\") pod \"maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q\" (UID: \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"
Oct 30 01:19:17.179960 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.179942    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txlkg\" (UniqueName: \"kubernetes.io/projected/1272be00-9aa6-4cb1-9d1e-74cfe0164569-kube-api-access-txlkg\") pod \"maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q\" (UID: \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"
Oct 30 01:19:17.180073 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.179957    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1272be00-9aa6-4cb1-9d1e-74cfe0164569-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q\" (UID: \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"
Oct 30 01:19:17.180197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.180172    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk9ss\" (UniqueName: \"kubernetes.io/projected/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-kube-api-access-mk9ss\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.180271 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.180231    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.180271 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.180249    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.180271 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.180265    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnbjk\" (UniqueName: \"kubernetes.io/projected/4f42de42-9772-41e0-9cdb-8a9b212106ee-kube-api-access-xnbjk\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.180441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.180280    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.180441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.180296    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.180441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.180311    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.180441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.180324    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.180441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.180337    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.180441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.180350    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.180441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.180366    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f42de42-9772-41e0-9cdb-8a9b212106ee-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.180441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.180380    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.180441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.180392    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f42de42-9772-41e0-9cdb-8a9b212106ee-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.180441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.180406    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a528dfda-e8fb-4a8e-abe7-4d870b94a64a-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:19:17.180790 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.180449    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a0acc3-77a4-4ad2-978e-08280840d903-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m\" (UID: \"57a0acc3-77a4-4ad2-978e-08280840d903\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m"
Oct 30 01:19:17.180790 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.180456    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261fed99-0c9e-42cb-ad91-540d2b125576-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b\" (UID: \"261fed99-0c9e-42cb-ad91-540d2b125576\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b"
Oct 30 01:19:17.182337 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.182317    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1272be00-9aa6-4cb1-9d1e-74cfe0164569-metrics-certs\") pod \"maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q\" (UID: \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"
Oct 30 01:19:17.182387 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.182335    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1272be00-9aa6-4cb1-9d1e-74cfe0164569-maas-api-tls\") pod \"maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q\" (UID: \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"
Oct 30 01:19:17.219787 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.219761    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9k7f\" (UniqueName: \"kubernetes.io/projected/57a0acc3-77a4-4ad2-978e-08280840d903-kube-api-access-x9k7f\") pod \"payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m\" (UID: \"57a0acc3-77a4-4ad2-978e-08280840d903\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m"
Oct 30 01:19:17.220422 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.220025    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl9fk\" (UniqueName: \"kubernetes.io/projected/261fed99-0c9e-42cb-ad91-540d2b125576-kube-api-access-gl9fk\") pod \"payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b\" (UID: \"261fed99-0c9e-42cb-ad91-540d2b125576\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b"
Oct 30 01:19:17.220422 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.220365    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txlkg\" (UniqueName: \"kubernetes.io/projected/1272be00-9aa6-4cb1-9d1e-74cfe0164569-kube-api-access-txlkg\") pod \"maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q\" (UID: \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"
Oct 30 01:19:17.222836 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.222813    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"]
Oct 30 01:19:17.226670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.226647    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-qmr9l"]
Oct 30 01:19:17.256176 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.256150    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m"
Oct 30 01:19:17.264610 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.264583    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"]
Oct 30 01:19:17.269254 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.269223    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-bjxp7"]
Oct 30 01:19:17.388490 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.388461    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b"
Oct 30 01:19:17.505932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.501529    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"
Oct 30 01:19:17.545228 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.545162    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"]
Oct 30 01:19:17.551607 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.550986    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-58ghs"]
Oct 30 01:19:17.582206 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.579345    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"]
Oct 30 01:19:17.587868 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.587803    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-320045-openshift-default-6dcdf94b5c-g4kjg"]
Oct 30 01:19:17.593661 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.593630    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m"]
Oct 30 01:19:17.720756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.720729    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b"]
Oct 30 01:19:17.721124 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:19:17.721102    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261fed99_0c9e_42cb_ad91_540d2b125576.slice/crio-cc4a73a624af7b6cde900f9ad529117a44c6023b530c037bde19052c12b9ef05 WatchSource:0}: Error finding container cc4a73a624af7b6cde900f9ad529117a44c6023b530c037bde19052c12b9ef05: Status 404 returned error can't find the container with id cc4a73a624af7b6cde900f9ad529117a44c6023b530c037bde19052c12b9ef05
Oct 30 01:19:17.834192 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:19:17.834138    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1272be00_9aa6_4cb1_9d1e_74cfe0164569.slice/crio-1e7be587a83560dade452ff32cee6d14871fa6f140af1da8798f29ecbc6603c6 WatchSource:0}: Error finding container 1e7be587a83560dade452ff32cee6d14871fa6f140af1da8798f29ecbc6603c6: Status 404 returned error can't find the container with id 1e7be587a83560dade452ff32cee6d14871fa6f140af1da8798f29ecbc6603c6
Oct 30 01:19:17.834333 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:17.834183    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"]
Oct 30 01:19:18.059397 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:18.059362    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b" event={"ID":"261fed99-0c9e-42cb-ad91-540d2b125576","Type":"ContainerStarted","Data":"adb8cba0c97eb094676379597f401d4072f38fcf60e2da07a7b5742880cf5a9b"}
Oct 30 01:19:18.059577 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:18.059413    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b" event={"ID":"261fed99-0c9e-42cb-ad91-540d2b125576","Type":"ContainerStarted","Data":"cc4a73a624af7b6cde900f9ad529117a44c6023b530c037bde19052c12b9ef05"}
Oct 30 01:19:18.059577 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:18.059455    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b"
Oct 30 01:19:18.061165 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:18.061135    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m" event={"ID":"57a0acc3-77a4-4ad2-978e-08280840d903","Type":"ContainerStarted","Data":"897ae639aa8f5e6562e39a7841af7bea83634e8c9ba584275e432d1c7c36f6f3"}
Oct 30 01:19:18.061295 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:18.061189    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m" event={"ID":"57a0acc3-77a4-4ad2-978e-08280840d903","Type":"ContainerStarted","Data":"e2a845639c73da83207d59ebddc821a72f3676c02318175268224d96c424b7c8"}
Oct 30 01:19:18.061295 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:18.061222    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m"
Oct 30 01:19:18.062541 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:18.062515    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q" event={"ID":"1272be00-9aa6-4cb1-9d1e-74cfe0164569","Type":"ContainerStarted","Data":"1e7be587a83560dade452ff32cee6d14871fa6f140af1da8798f29ecbc6603c6"}
Oct 30 01:19:18.101519 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:18.101461    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b" podStartSLOduration=112.101443483 podStartE2EDuration="1m52.101443483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:18.100384362 +0000 UTC m=+2651.957292117" watchObservedRunningTime="2026-10-30 01:19:18.101443483 +0000 UTC m=+2651.958351238"
Oct 30 01:19:18.143531 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:18.143463    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m" podStartSLOduration=101.143445535 podStartE2EDuration="1m41.143445535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:18.142372157 +0000 UTC m=+2651.999279911" watchObservedRunningTime="2026-10-30 01:19:18.143445535 +0000 UTC m=+2652.000353291"
Oct 30 01:19:18.315134 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:18.315053    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49482cc2-07e6-455e-952e-bd42810e8d7b" path="/var/lib/kubelet/pods/49482cc2-07e6-455e-952e-bd42810e8d7b/volumes"
Oct 30 01:19:18.315798 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:18.315775    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f42de42-9772-41e0-9cdb-8a9b212106ee" path="/var/lib/kubelet/pods/4f42de42-9772-41e0-9cdb-8a9b212106ee/volumes"
Oct 30 01:19:18.316485 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:18.316466    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a528dfda-e8fb-4a8e-abe7-4d870b94a64a" path="/var/lib/kubelet/pods/a528dfda-e8fb-4a8e-abe7-4d870b94a64a/volumes"
Oct 30 01:19:18.317544 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:18.317524    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc28ff31-b61a-4d8e-9ff3-361345eb9c06" path="/var/lib/kubelet/pods/cc28ff31-b61a-4d8e-9ff3-361345eb9c06/volumes"
Oct 30 01:19:19.169018 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:19.168977    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q" event={"ID":"1272be00-9aa6-4cb1-9d1e-74cfe0164569","Type":"ContainerStarted","Data":"7a69e9c3ef70c5d906be29da9edd8ceb7846dbdb49dd0da2c50cda004103d597"}
Oct 30 01:19:19.169665 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:19.169643    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"
Oct 30 01:19:19.215751 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:19.215690    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q" podStartSLOduration=83.891144655 podStartE2EDuration="1m24.215677276s" totalImagesPullingTime="324.532621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:19.210681604 +0000 UTC m=+2653.067589364" watchObservedRunningTime="2026-10-30 01:19:19.215677276 +0000 UTC m=+2653.072585031"
Oct 30 01:19:25.520025 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:25.519986    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:19:26.205463 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:26.205427    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"
Oct 30 01:19:29.169830 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:29.169797    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m"
Oct 30 01:19:29.171239 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:29.171216    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b"
Oct 30 01:19:35.119772 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.119354    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"]
Oct 30 01:19:35.134379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.133262    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.154436 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.152425    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6cba49-openshift-default-dockercfg-mdk9j\""
Oct 30 01:19:35.229068 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.228453    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"]
Oct 30 01:19:35.230956 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.230392    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-workload-certs\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.230956 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.230460    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r77d4\" (UniqueName: \"kubernetes.io/projected/51e427c9-5c89-4401-afa6-437676618dbf-kube-api-access-r77d4\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.230956 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.230576    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-istio-data\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.230956 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.230616    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-credential-socket\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.230956 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.230698    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e427c9-5c89-4401-afa6-437676618dbf-istiod-ca-cert\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.230956 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.230790    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e427c9-5c89-4401-afa6-437676618dbf-istio-podinfo\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.232110 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.231407    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-istio-envoy\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.232110 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.231607    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-workload-socket\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.232110 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.231690    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e427c9-5c89-4401-afa6-437676618dbf-istio-token\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.414148 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.414048    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e427c9-5c89-4401-afa6-437676618dbf-istio-podinfo\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.436737 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.431590    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-istio-envoy\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.436737 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.431837    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-workload-socket\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.436737 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.431942    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e427c9-5c89-4401-afa6-437676618dbf-istio-token\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.436737 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.432036    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-workload-certs\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.436737 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.432077    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r77d4\" (UniqueName: \"kubernetes.io/projected/51e427c9-5c89-4401-afa6-437676618dbf-kube-api-access-r77d4\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.436737 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.432199    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-istio-data\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.436737 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.432244    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-credential-socket\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.436737 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.432314    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e427c9-5c89-4401-afa6-437676618dbf-istiod-ca-cert\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.436737 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.433195    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e427c9-5c89-4401-afa6-437676618dbf-istiod-ca-cert\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.436737 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.434038    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-workload-socket\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.436737 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.434740    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-workload-certs\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.436737 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.435168    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-istio-data\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.436737 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.435394    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-credential-socket\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.460778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.448012    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e427c9-5c89-4401-afa6-437676618dbf-istio-podinfo\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.460778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.448313    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-istio-envoy\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.475748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.473347    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e427c9-5c89-4401-afa6-437676618dbf-istio-token\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.478735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.476279    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r77d4\" (UniqueName: \"kubernetes.io/projected/51e427c9-5c89-4401-afa6-437676618dbf-kube-api-access-r77d4\") pod \"e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") " pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.554863 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.554428    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:35.881578 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:19:35.881548    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e427c9_5c89_4401_afa6_437676618dbf.slice/crio-431a4081ca0e21f565a1d8a311b3db9533881b6114cbbe88b6e85c9f4b580c99 WatchSource:0}: Error finding container 431a4081ca0e21f565a1d8a311b3db9533881b6114cbbe88b6e85c9f4b580c99: Status 404 returned error can't find the container with id 431a4081ca0e21f565a1d8a311b3db9533881b6114cbbe88b6e85c9f4b580c99
Oct 30 01:19:35.881903 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.881884    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"]
Oct 30 01:19:35.884560 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.884522    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:19:35.884675 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.884600    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:19:35.884675 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:35.884649    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:19:36.443801 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:36.443765    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp" event={"ID":"51e427c9-5c89-4401-afa6-437676618dbf","Type":"ContainerStarted","Data":"509c1ad84607be8d861219d828745769d199bccd3408f168903ad43a347af7d8"}
Oct 30 01:19:36.444187 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:36.443830    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp" event={"ID":"51e427c9-5c89-4401-afa6-437676618dbf","Type":"ContainerStarted","Data":"431a4081ca0e21f565a1d8a311b3db9533881b6114cbbe88b6e85c9f4b580c99"}
Oct 30 01:19:36.490745 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:36.490681    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp" podStartSLOduration=1.490667649 podStartE2EDuration="1.490667649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:36.48704445 +0000 UTC m=+2670.343952205" watchObservedRunningTime="2026-10-30 01:19:36.490667649 +0000 UTC m=+2670.347575403"
Oct 30 01:19:36.554638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:36.554600    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:36.560555 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:36.560530    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:37.488787 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:37.488615    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:37.501280 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:19:37.500951    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:19:46.630821 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:19:46.630781    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:20:05.664982 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:05.664947    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx"]
Oct 30 01:20:05.696754 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:05.696719    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s"]
Oct 30 01:20:06.877617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:06.877589    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 01:20:06.891310 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:06.891288    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 01:20:09.851794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:09.851739    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"]
Oct 30 01:20:09.852333 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:09.851959    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7" podUID="7fb31e16-f069-4549-ab6b-c8cdb62b43aa" containerName="maas-api" containerID="cri-o://80f91f36be7d5ed36785d19b031597ee600a2de93a5b2f5adc7d7ca026b987d3" gracePeriod=30
Oct 30 01:20:10.157743 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.157701    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:20:10.208420 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.208385    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-kube-api-access-qmt6j\" (UniqueName: \"kubernetes.io/projected/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-kube-api-access-qmt6j\") pod \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\" (UID: \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\") "
Oct 30 01:20:10.208585 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.208455    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-openshift-service-ca\") pod \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\" (UID: \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\") "
Oct 30 01:20:10.208585 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.208535    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-metrics-certs\") pod \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\" (UID: \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\") "
Oct 30 01:20:10.208674 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.208606    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-maas-api-tls\") pod \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\" (UID: \"7fb31e16-f069-4549-ab6b-c8cdb62b43aa\") "
Oct 30 01:20:10.210066 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.210039    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-openshift-service-ca" pod "7fb31e16-f069-4549-ab6b-c8cdb62b43aa" (UID: "7fb31e16-f069-4549-ab6b-c8cdb62b43aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:10.211950 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.211920    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-maas-api-tls" pod "7fb31e16-f069-4549-ab6b-c8cdb62b43aa" (UID: "7fb31e16-f069-4549-ab6b-c8cdb62b43aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:10.212120 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.212099    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-kube-api-access-qmt6j" pod "7fb31e16-f069-4549-ab6b-c8cdb62b43aa" (UID: "7fb31e16-f069-4549-ab6b-c8cdb62b43aa"). InnerVolumeSpecName "kube-api-access-qmt6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:10.212170 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.212106    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-metrics-certs" pod "7fb31e16-f069-4549-ab6b-c8cdb62b43aa" (UID: "7fb31e16-f069-4549-ab6b-c8cdb62b43aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:10.311588 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.311550    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:10.311761 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.311591    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmt6j\" (UniqueName: \"kubernetes.io/projected/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-kube-api-access-qmt6j\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:10.311761 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.311606    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:10.311761 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.311619    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fb31e16-f069-4549-ab6b-c8cdb62b43aa-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:10.666228 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.666171    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx" podUID="a003430b-13be-47b6-864c-9062ae70f676" containerName="payload-processing" containerID="cri-o://307aba3e165a282d1d9ec24ecb2f268308dea98948b0cf9e8058c95792938335" gracePeriod=25
Oct 30 01:20:10.697773 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.697739    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s" podUID="5eb2c17f-f68a-4672-b266-f3562154baa3" containerName="payload-pre-processing" containerID="cri-o://347b93dfefbcdc70315113b5e4bc91862a2a1406f877c10398a06a42761a823f" gracePeriod=25
Oct 30 01:20:10.838751 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.838701    6099 generic.go:356] "Generic (PLEG): container finished" podID="7fb31e16-f069-4549-ab6b-c8cdb62b43aa" containerID="80f91f36be7d5ed36785d19b031597ee600a2de93a5b2f5adc7d7ca026b987d3" exitCode=0
Oct 30 01:20:10.838751 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.838735    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7" event={"ID":"7fb31e16-f069-4549-ab6b-c8cdb62b43aa","Type":"ContainerDied","Data":"80f91f36be7d5ed36785d19b031597ee600a2de93a5b2f5adc7d7ca026b987d3"}
Oct 30 01:20:10.839021 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.838759    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"
Oct 30 01:20:10.839021 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.838800    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7" event={"ID":"7fb31e16-f069-4549-ab6b-c8cdb62b43aa","Type":"ContainerDied","Data":"58562a4494edf45e1f72bd7cf1987f5bbb72e36a209647132122733d574ccf2a"}
Oct 30 01:20:10.839021 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.838828    6099 scope.go:122] "RemoveContainer" containerID="80f91f36be7d5ed36785d19b031597ee600a2de93a5b2f5adc7d7ca026b987d3"
Oct 30 01:20:10.850919 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.850897    6099 scope.go:122] "RemoveContainer" containerID="80f91f36be7d5ed36785d19b031597ee600a2de93a5b2f5adc7d7ca026b987d3"
Oct 30 01:20:10.851223 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:20:10.851194    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f91f36be7d5ed36785d19b031597ee600a2de93a5b2f5adc7d7ca026b987d3\": container with ID starting with 80f91f36be7d5ed36785d19b031597ee600a2de93a5b2f5adc7d7ca026b987d3 not found: ID does not exist" containerID="80f91f36be7d5ed36785d19b031597ee600a2de93a5b2f5adc7d7ca026b987d3"
Oct 30 01:20:10.851346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.851229    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f91f36be7d5ed36785d19b031597ee600a2de93a5b2f5adc7d7ca026b987d3"} err="failed to get container status \"80f91f36be7d5ed36785d19b031597ee600a2de93a5b2f5adc7d7ca026b987d3\": rpc error: code = NotFound desc = could not find container \"80f91f36be7d5ed36785d19b031597ee600a2de93a5b2f5adc7d7ca026b987d3\": container with ID starting with 80f91f36be7d5ed36785d19b031597ee600a2de93a5b2f5adc7d7ca026b987d3 not found: ID does not exist"
Oct 30 01:20:10.891402 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.891372    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"]
Oct 30 01:20:10.895639 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:10.895614    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f69104-5b86bc7557-9kxc7"]
Oct 30 01:20:12.264897 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:12.264866    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fb31e16-f069-4549-ab6b-c8cdb62b43aa" path="/var/lib/kubelet/pods/7fb31e16-f069-4549-ab6b-c8cdb62b43aa/volumes"
Oct 30 01:20:13.311595 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:13.311556    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx" podUID="a003430b-13be-47b6-864c-9062ae70f676" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:20:13.311991 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:13.311556    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s" podUID="5eb2c17f-f68a-4672-b266-f3562154baa3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:20:16.310089 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:16.310054    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:20:16.632922 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:20:16.632837    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:20:18.097925 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:18.097837    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"]
Oct 30 01:20:18.111229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:18.111083    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss" podUID="f0bbb319-2301-41ef-879f-29ea51d8699e" containerName="istio-proxy" containerID="cri-o://27a204116bf3ae3983e66ccc6968a3b568297d72a9a11ce458ebf65836f5621a" gracePeriod=30
Oct 30 01:20:18.485783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:18.483227    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:18.546390 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:18.546317    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:18.581740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:18.577977    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"]
Oct 30 01:20:18.581740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:18.578290    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz" podUID="e2c83697-c779-4359-804d-b6b4b4d5e77c" containerName="istio-proxy" containerID="cri-o://ea435deb2d0c40088ab9a8815b3d6f6a76b4cb8e74a297a9ecf99118259db5e6" gracePeriod=30
Oct 30 01:20:18.581740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:18.580051    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded"
Oct 30 01:20:18.756899 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:18.756805    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:20:22.001243 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:22.001207    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"]
Oct 30 01:20:22.001678 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:22.001440    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m" podUID="d8655eb2-c236-49f0-90e8-b7b4ee6e51be" containerName="istio-proxy" containerID="cri-o://b503fc5080acd83abef3b4b7395e0ce885bde57e2d7accb4659ee5b7ab24faaf" gracePeriod=30
Oct 30 01:20:23.312305 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:23.312259    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s" podUID="5eb2c17f-f68a-4672-b266-f3562154baa3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:20:23.312973 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:23.312377    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx" podUID="a003430b-13be-47b6-864c-9062ae70f676" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:20:23.555387 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:23.555352    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"]
Oct 30 01:20:23.953190 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:23.953161    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:20:23.995616 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:23.995570    6099 generic.go:356] "Generic (PLEG): container finished" podID="f0bbb319-2301-41ef-879f-29ea51d8699e" containerID="27a204116bf3ae3983e66ccc6968a3b568297d72a9a11ce458ebf65836f5621a" exitCode=0
Oct 30 01:20:23.995819 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:23.995654    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss" event={"ID":"f0bbb319-2301-41ef-879f-29ea51d8699e","Type":"ContainerDied","Data":"27a204116bf3ae3983e66ccc6968a3b568297d72a9a11ce458ebf65836f5621a"}
Oct 30 01:20:23.998675 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:23.998653    6099 generic.go:356] "Generic (PLEG): container finished" podID="e2c83697-c779-4359-804d-b6b4b4d5e77c" containerID="ea435deb2d0c40088ab9a8815b3d6f6a76b4cb8e74a297a9ecf99118259db5e6" exitCode=0
Oct 30 01:20:23.998860 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:23.998733    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz" event={"ID":"e2c83697-c779-4359-804d-b6b4b4d5e77c","Type":"ContainerDied","Data":"ea435deb2d0c40088ab9a8815b3d6f6a76b4cb8e74a297a9ecf99118259db5e6"}
Oct 30 01:20:23.998860 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:23.998783    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz" event={"ID":"e2c83697-c779-4359-804d-b6b4b4d5e77c","Type":"ContainerDied","Data":"ae336b57e4202ba0f18986d534ca3a7c2e72d4a27317add78bcf4dab4def721a"}
Oct 30 01:20:23.998860 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:23.998799    6099 scope.go:122] "RemoveContainer" containerID="ea435deb2d0c40088ab9a8815b3d6f6a76b4cb8e74a297a9ecf99118259db5e6"
Oct 30 01:20:23.998860 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:23.998833    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"
Oct 30 01:20:24.017283 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.017260    6099 scope.go:122] "RemoveContainer" containerID="1f43de5288b1add050adc94449f17f9bcfbad724425c6e912e546a11f767230c"
Oct 30 01:20:24.039729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.039049    6099 scope.go:122] "RemoveContainer" containerID="ea435deb2d0c40088ab9a8815b3d6f6a76b4cb8e74a297a9ecf99118259db5e6"
Oct 30 01:20:24.039729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.039467    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-envoy\") pod \"e2c83697-c779-4359-804d-b6b4b4d5e77c\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") "
Oct 30 01:20:24.039729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.039529    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-credential-socket\") pod \"e2c83697-c779-4359-804d-b6b4b4d5e77c\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") "
Oct 30 01:20:24.039989 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:20:24.039472    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea435deb2d0c40088ab9a8815b3d6f6a76b4cb8e74a297a9ecf99118259db5e6\": container with ID starting with ea435deb2d0c40088ab9a8815b3d6f6a76b4cb8e74a297a9ecf99118259db5e6 not found: ID does not exist" containerID="ea435deb2d0c40088ab9a8815b3d6f6a76b4cb8e74a297a9ecf99118259db5e6"
Oct 30 01:20:24.039989 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.039832    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea435deb2d0c40088ab9a8815b3d6f6a76b4cb8e74a297a9ecf99118259db5e6"} err="failed to get container status \"ea435deb2d0c40088ab9a8815b3d6f6a76b4cb8e74a297a9ecf99118259db5e6\": rpc error: code = NotFound desc = could not find container \"ea435deb2d0c40088ab9a8815b3d6f6a76b4cb8e74a297a9ecf99118259db5e6\": container with ID starting with ea435deb2d0c40088ab9a8815b3d6f6a76b4cb8e74a297a9ecf99118259db5e6 not found: ID does not exist"
Oct 30 01:20:24.039989 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.039861    6099 scope.go:122] "RemoveContainer" containerID="1f43de5288b1add050adc94449f17f9bcfbad724425c6e912e546a11f767230c"
Oct 30 01:20:24.040180 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.040128    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-credential-socket" pod "e2c83697-c779-4359-804d-b6b4b4d5e77c" (UID: "e2c83697-c779-4359-804d-b6b4b4d5e77c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:24.040644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.040283    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2c83697-c779-4359-804d-b6b4b4d5e77c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2c83697-c779-4359-804d-b6b4b4d5e77c-istiod-ca-cert\") pod \"e2c83697-c779-4359-804d-b6b4b4d5e77c\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") "
Oct 30 01:20:24.040644 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:20:24.040290    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f43de5288b1add050adc94449f17f9bcfbad724425c6e912e546a11f767230c\": container with ID starting with 1f43de5288b1add050adc94449f17f9bcfbad724425c6e912e546a11f767230c not found: ID does not exist" containerID="1f43de5288b1add050adc94449f17f9bcfbad724425c6e912e546a11f767230c"
Oct 30 01:20:24.040644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.040324    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f43de5288b1add050adc94449f17f9bcfbad724425c6e912e546a11f767230c"} err="failed to get container status \"1f43de5288b1add050adc94449f17f9bcfbad724425c6e912e546a11f767230c\": rpc error: code = NotFound desc = could not find container \"1f43de5288b1add050adc94449f17f9bcfbad724425c6e912e546a11f767230c\": container with ID starting with 1f43de5288b1add050adc94449f17f9bcfbad724425c6e912e546a11f767230c not found: ID does not exist"
Oct 30 01:20:24.040644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.040329    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2c83697-c779-4359-804d-b6b4b4d5e77c-kube-api-access-mfs4x\" (UniqueName: \"kubernetes.io/projected/e2c83697-c779-4359-804d-b6b4b4d5e77c-kube-api-access-mfs4x\") pod \"e2c83697-c779-4359-804d-b6b4b4d5e77c\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") "
Oct 30 01:20:24.040644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.040449    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-data\") pod \"e2c83697-c779-4359-804d-b6b4b4d5e77c\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") "
Oct 30 01:20:24.040644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.040491    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-token\" (UniqueName: \"kubernetes.io/projected/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-token\") pod \"e2c83697-c779-4359-804d-b6b4b4d5e77c\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") "
Oct 30 01:20:24.040644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.040519    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-workload-socket\") pod \"e2c83697-c779-4359-804d-b6b4b4d5e77c\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") "
Oct 30 01:20:24.040644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.040565    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-workload-certs\") pod \"e2c83697-c779-4359-804d-b6b4b4d5e77c\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") "
Oct 30 01:20:24.041244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.040730    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-podinfo\") pod \"e2c83697-c779-4359-804d-b6b4b4d5e77c\" (UID: \"e2c83697-c779-4359-804d-b6b4b4d5e77c\") "
Oct 30 01:20:24.041244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.040815    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2c83697-c779-4359-804d-b6b4b4d5e77c-istiod-ca-cert" pod "e2c83697-c779-4359-804d-b6b4b4d5e77c" (UID: "e2c83697-c779-4359-804d-b6b4b4d5e77c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:24.041445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.041421    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-workload-socket" pod "e2c83697-c779-4359-804d-b6b4b4d5e77c" (UID: "e2c83697-c779-4359-804d-b6b4b4d5e77c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:24.042006 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.041982    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-data" pod "e2c83697-c779-4359-804d-b6b4b4d5e77c" (UID: "e2c83697-c779-4359-804d-b6b4b4d5e77c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:24.042111 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.042076    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.042111 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.042101    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2c83697-c779-4359-804d-b6b4b4d5e77c-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.042242 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.042116    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.042619 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.042592    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-workload-certs" pod "e2c83697-c779-4359-804d-b6b4b4d5e77c" (UID: "e2c83697-c779-4359-804d-b6b4b4d5e77c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:24.044117 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.044090    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c83697-c779-4359-804d-b6b4b4d5e77c-kube-api-access-mfs4x" pod "e2c83697-c779-4359-804d-b6b4b4d5e77c" (UID: "e2c83697-c779-4359-804d-b6b4b4d5e77c"). InnerVolumeSpecName "kube-api-access-mfs4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:24.044601 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.044574    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-envoy" pod "e2c83697-c779-4359-804d-b6b4b4d5e77c" (UID: "e2c83697-c779-4359-804d-b6b4b4d5e77c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:24.045198 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.045168    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-podinfo" pod "e2c83697-c779-4359-804d-b6b4b4d5e77c" (UID: "e2c83697-c779-4359-804d-b6b4b4d5e77c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:20:24.045675 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.045652    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-token" pod "e2c83697-c779-4359-804d-b6b4b4d5e77c" (UID: "e2c83697-c779-4359-804d-b6b4b4d5e77c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:24.077816 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.077793    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:20:24.145317 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.145276    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-token\" (UniqueName: \"kubernetes.io/projected/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-token\") pod \"f0bbb319-2301-41ef-879f-29ea51d8699e\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") "
Oct 30 01:20:24.145517 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.145333    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-workload-socket\") pod \"f0bbb319-2301-41ef-879f-29ea51d8699e\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") "
Oct 30 01:20:24.145517 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.145451    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-data\") pod \"f0bbb319-2301-41ef-879f-29ea51d8699e\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") "
Oct 30 01:20:24.145517 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.145489    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-credential-socket\") pod \"f0bbb319-2301-41ef-879f-29ea51d8699e\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") "
Oct 30 01:20:24.145773 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.145623    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-envoy\") pod \"f0bbb319-2301-41ef-879f-29ea51d8699e\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") "
Oct 30 01:20:24.145773 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.145665    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-workload-certs\") pod \"f0bbb319-2301-41ef-879f-29ea51d8699e\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") "
Oct 30 01:20:24.145773 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.145660    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-workload-socket" pod "f0bbb319-2301-41ef-879f-29ea51d8699e" (UID: "f0bbb319-2301-41ef-879f-29ea51d8699e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:24.145910 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.145794    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0bbb319-2301-41ef-879f-29ea51d8699e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0bbb319-2301-41ef-879f-29ea51d8699e-istiod-ca-cert\") pod \"f0bbb319-2301-41ef-879f-29ea51d8699e\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") "
Oct 30 01:20:24.146007 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.145941    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-workload-certs" pod "f0bbb319-2301-41ef-879f-29ea51d8699e" (UID: "f0bbb319-2301-41ef-879f-29ea51d8699e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:24.146007 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.145997    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-podinfo\") pod \"f0bbb319-2301-41ef-879f-29ea51d8699e\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") "
Oct 30 01:20:24.146105 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.146084    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0bbb319-2301-41ef-879f-29ea51d8699e-kube-api-access-7r7bt\" (UniqueName: \"kubernetes.io/projected/f0bbb319-2301-41ef-879f-29ea51d8699e-kube-api-access-7r7bt\") pod \"f0bbb319-2301-41ef-879f-29ea51d8699e\" (UID: \"f0bbb319-2301-41ef-879f-29ea51d8699e\") "
Oct 30 01:20:24.146184 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.146137    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-data" pod "f0bbb319-2301-41ef-879f-29ea51d8699e" (UID: "f0bbb319-2301-41ef-879f-29ea51d8699e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:24.146360 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.146334    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-credential-socket" pod "f0bbb319-2301-41ef-879f-29ea51d8699e" (UID: "f0bbb319-2301-41ef-879f-29ea51d8699e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:24.146513 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.146486    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bbb319-2301-41ef-879f-29ea51d8699e-istiod-ca-cert" pod "f0bbb319-2301-41ef-879f-29ea51d8699e" (UID: "f0bbb319-2301-41ef-879f-29ea51d8699e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:24.147363 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.147335    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.147486 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.147370    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfs4x\" (UniqueName: \"kubernetes.io/projected/e2c83697-c779-4359-804d-b6b4b4d5e77c-kube-api-access-mfs4x\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.147486 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.147390    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.147486 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.147417    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.147486 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.147431    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.147486 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.147444    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.147486 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.147459    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.147486 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.147477    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2c83697-c779-4359-804d-b6b4b4d5e77c-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.147486 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.147488    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2c83697-c779-4359-804d-b6b4b4d5e77c-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.148036 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.147501    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.148036 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.147515    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0bbb319-2301-41ef-879f-29ea51d8699e-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.148183 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.148144    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-token" pod "f0bbb319-2301-41ef-879f-29ea51d8699e" (UID: "f0bbb319-2301-41ef-879f-29ea51d8699e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:24.149275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.149251    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bbb319-2301-41ef-879f-29ea51d8699e-kube-api-access-7r7bt" pod "f0bbb319-2301-41ef-879f-29ea51d8699e" (UID: "f0bbb319-2301-41ef-879f-29ea51d8699e"). InnerVolumeSpecName "kube-api-access-7r7bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:24.149435 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.149293    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-envoy" pod "f0bbb319-2301-41ef-879f-29ea51d8699e" (UID: "f0bbb319-2301-41ef-879f-29ea51d8699e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:24.149667 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.149648    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-podinfo" pod "f0bbb319-2301-41ef-879f-29ea51d8699e" (UID: "f0bbb319-2301-41ef-879f-29ea51d8699e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:20:24.248787 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.248674    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.248787 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.248715    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.248787 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.248724    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0bbb319-2301-41ef-879f-29ea51d8699e-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.248787 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.248735    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r7bt\" (UniqueName: \"kubernetes.io/projected/f0bbb319-2301-41ef-879f-29ea51d8699e-kube-api-access-7r7bt\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:24.339530 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.339500    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"]
Oct 30 01:20:24.345773 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:24.345746    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-cdglz"]
Oct 30 01:20:25.007552 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:25.007473    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"
Oct 30 01:20:25.007893 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:25.007473    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss" event={"ID":"f0bbb319-2301-41ef-879f-29ea51d8699e","Type":"ContainerDied","Data":"c30d654a412b5da137f7f2522113e1af04ac227f5dd85984868150827a2ba7e0"}
Oct 30 01:20:25.007893 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:25.007626    6099 scope.go:122] "RemoveContainer" containerID="27a204116bf3ae3983e66ccc6968a3b568297d72a9a11ce458ebf65836f5621a"
Oct 30 01:20:25.061371 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:25.061348    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"]
Oct 30 01:20:25.072969 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:25.072943    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f69104-openshift-default-5c7f96bd7-wx5ss"]
Oct 30 01:20:26.266357 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:26.266324    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2c83697-c779-4359-804d-b6b4b4d5e77c" path="/var/lib/kubelet/pods/e2c83697-c779-4359-804d-b6b4b4d5e77c/volumes"
Oct 30 01:20:26.266945 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:26.266927    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0bbb319-2301-41ef-879f-29ea51d8699e" path="/var/lib/kubelet/pods/f0bbb319-2301-41ef-879f-29ea51d8699e/volumes"
Oct 30 01:20:27.371925 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.371901    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:20:27.499552 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.499515    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istiod-ca-cert\") pod \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") "
Oct 30 01:20:27.499756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.499715    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-kube-api-access-7hqkd\" (UniqueName: \"kubernetes.io/projected/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-kube-api-access-7hqkd\") pod \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") "
Oct 30 01:20:27.499823 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.499767    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-data\") pod \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") "
Oct 30 01:20:27.499823 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.499791    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-token\" (UniqueName: \"kubernetes.io/projected/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-token\") pod \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") "
Oct 30 01:20:27.499942 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.499909    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istiod-ca-cert" pod "d8655eb2-c236-49f0-90e8-b7b4ee6e51be" (UID: "d8655eb2-c236-49f0-90e8-b7b4ee6e51be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:27.500003 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.499945    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-workload-certs\") pod \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") "
Oct 30 01:20:27.500060 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.500006    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-workload-socket\") pod \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") "
Oct 30 01:20:27.500060 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.500049    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-envoy\") pod \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") "
Oct 30 01:20:27.500181 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.500065    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-data" pod "d8655eb2-c236-49f0-90e8-b7b4ee6e51be" (UID: "d8655eb2-c236-49f0-90e8-b7b4ee6e51be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:27.500181 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.500115    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-podinfo\") pod \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") "
Oct 30 01:20:27.500181 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.500175    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-credential-socket\") pod \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\" (UID: \"d8655eb2-c236-49f0-90e8-b7b4ee6e51be\") "
Oct 30 01:20:27.500350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.500255    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-workload-certs" pod "d8655eb2-c236-49f0-90e8-b7b4ee6e51be" (UID: "d8655eb2-c236-49f0-90e8-b7b4ee6e51be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:27.500350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.500280    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-workload-socket" pod "d8655eb2-c236-49f0-90e8-b7b4ee6e51be" (UID: "d8655eb2-c236-49f0-90e8-b7b4ee6e51be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:27.501967 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.500540    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-credential-socket" pod "d8655eb2-c236-49f0-90e8-b7b4ee6e51be" (UID: "d8655eb2-c236-49f0-90e8-b7b4ee6e51be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:27.501967 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.501189    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:27.501967 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.501216    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:27.501967 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.501231    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:27.501967 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.501244    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:27.501967 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.501258    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:27.502481 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.502461    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-kube-api-access-7hqkd" pod "d8655eb2-c236-49f0-90e8-b7b4ee6e51be" (UID: "d8655eb2-c236-49f0-90e8-b7b4ee6e51be"). InnerVolumeSpecName "kube-api-access-7hqkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:27.502641 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.502619    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-token" pod "d8655eb2-c236-49f0-90e8-b7b4ee6e51be" (UID: "d8655eb2-c236-49f0-90e8-b7b4ee6e51be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:27.502755 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.502692    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-podinfo" pod "d8655eb2-c236-49f0-90e8-b7b4ee6e51be" (UID: "d8655eb2-c236-49f0-90e8-b7b4ee6e51be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:20:27.502806 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.502760    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-envoy" pod "d8655eb2-c236-49f0-90e8-b7b4ee6e51be" (UID: "d8655eb2-c236-49f0-90e8-b7b4ee6e51be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:27.603397 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.603317    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:27.603397 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.603346    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:27.603397 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.603360    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hqkd\" (UniqueName: \"kubernetes.io/projected/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-kube-api-access-7hqkd\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:27.603397 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:27.603373    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8655eb2-c236-49f0-90e8-b7b4ee6e51be-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:28.036968 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:28.036888    6099 generic.go:356] "Generic (PLEG): container finished" podID="d8655eb2-c236-49f0-90e8-b7b4ee6e51be" containerID="b503fc5080acd83abef3b4b7395e0ce885bde57e2d7accb4659ee5b7ab24faaf" exitCode=0
Oct 30 01:20:28.036968 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:28.036945    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"
Oct 30 01:20:28.037173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:28.036965    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m" event={"ID":"d8655eb2-c236-49f0-90e8-b7b4ee6e51be","Type":"ContainerDied","Data":"b503fc5080acd83abef3b4b7395e0ce885bde57e2d7accb4659ee5b7ab24faaf"}
Oct 30 01:20:28.037173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:28.037032    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m" event={"ID":"d8655eb2-c236-49f0-90e8-b7b4ee6e51be","Type":"ContainerDied","Data":"8c299ccd5347c8b4048098d0918125326aad89dd2eb970aabbbef36a950f55da"}
Oct 30 01:20:28.037173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:28.037063    6099 scope.go:122] "RemoveContainer" containerID="b503fc5080acd83abef3b4b7395e0ce885bde57e2d7accb4659ee5b7ab24faaf"
Oct 30 01:20:28.049295 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:28.049279    6099 scope.go:122] "RemoveContainer" containerID="b75dd08ca618e2049d074d706ebdf6720db7fd88718488d2f74f404fdb7c5c66"
Oct 30 01:20:28.060338 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:28.060309    6099 scope.go:122] "RemoveContainer" containerID="b503fc5080acd83abef3b4b7395e0ce885bde57e2d7accb4659ee5b7ab24faaf"
Oct 30 01:20:28.060632 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:20:28.060613    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b503fc5080acd83abef3b4b7395e0ce885bde57e2d7accb4659ee5b7ab24faaf\": container with ID starting with b503fc5080acd83abef3b4b7395e0ce885bde57e2d7accb4659ee5b7ab24faaf not found: ID does not exist" containerID="b503fc5080acd83abef3b4b7395e0ce885bde57e2d7accb4659ee5b7ab24faaf"
Oct 30 01:20:28.060690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:28.060640    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b503fc5080acd83abef3b4b7395e0ce885bde57e2d7accb4659ee5b7ab24faaf"} err="failed to get container status \"b503fc5080acd83abef3b4b7395e0ce885bde57e2d7accb4659ee5b7ab24faaf\": rpc error: code = NotFound desc = could not find container \"b503fc5080acd83abef3b4b7395e0ce885bde57e2d7accb4659ee5b7ab24faaf\": container with ID starting with b503fc5080acd83abef3b4b7395e0ce885bde57e2d7accb4659ee5b7ab24faaf not found: ID does not exist"
Oct 30 01:20:28.060690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:28.060657    6099 scope.go:122] "RemoveContainer" containerID="b75dd08ca618e2049d074d706ebdf6720db7fd88718488d2f74f404fdb7c5c66"
Oct 30 01:20:28.060951 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:20:28.060933    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b75dd08ca618e2049d074d706ebdf6720db7fd88718488d2f74f404fdb7c5c66\": container with ID starting with b75dd08ca618e2049d074d706ebdf6720db7fd88718488d2f74f404fdb7c5c66 not found: ID does not exist" containerID="b75dd08ca618e2049d074d706ebdf6720db7fd88718488d2f74f404fdb7c5c66"
Oct 30 01:20:28.061027 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:28.060956    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75dd08ca618e2049d074d706ebdf6720db7fd88718488d2f74f404fdb7c5c66"} err="failed to get container status \"b75dd08ca618e2049d074d706ebdf6720db7fd88718488d2f74f404fdb7c5c66\": rpc error: code = NotFound desc = could not find container \"b75dd08ca618e2049d074d706ebdf6720db7fd88718488d2f74f404fdb7c5c66\": container with ID starting with b75dd08ca618e2049d074d706ebdf6720db7fd88718488d2f74f404fdb7c5c66 not found: ID does not exist"
Oct 30 01:20:28.090219 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:28.090195    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"]
Oct 30 01:20:28.101863 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:28.101840    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f91e87-openshift-default-6d7b498dff-zgj9m"]
Oct 30 01:20:28.270583 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:28.270541    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8655eb2-c236-49f0-90e8-b7b4ee6e51be" path="/var/lib/kubelet/pods/d8655eb2-c236-49f0-90e8-b7b4ee6e51be/volumes"
Oct 30 01:20:31.060334 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.060158    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx"
Oct 30 01:20:31.097400 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.097372    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s"
Oct 30 01:20:31.120949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.120918    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb2c17f-f68a-4672-b266-f3562154baa3-kube-api-access-lbdkj\" (UniqueName: \"kubernetes.io/projected/5eb2c17f-f68a-4672-b266-f3562154baa3-kube-api-access-lbdkj\") pod \"5eb2c17f-f68a-4672-b266-f3562154baa3\" (UID: \"5eb2c17f-f68a-4672-b266-f3562154baa3\") "
Oct 30 01:20:31.121252 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.121215    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a003430b-13be-47b6-864c-9062ae70f676-kube-api-access-nmf4b\" (UniqueName: \"kubernetes.io/projected/a003430b-13be-47b6-864c-9062ae70f676-kube-api-access-nmf4b\") pod \"a003430b-13be-47b6-864c-9062ae70f676\" (UID: \"a003430b-13be-47b6-864c-9062ae70f676\") "
Oct 30 01:20:31.121509 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.121489    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a003430b-13be-47b6-864c-9062ae70f676-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a003430b-13be-47b6-864c-9062ae70f676-plugins-config-volume\") pod \"a003430b-13be-47b6-864c-9062ae70f676\" (UID: \"a003430b-13be-47b6-864c-9062ae70f676\") "
Oct 30 01:20:31.121792 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.121758    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb2c17f-f68a-4672-b266-f3562154baa3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb2c17f-f68a-4672-b266-f3562154baa3-plugins-config-volume\") pod \"5eb2c17f-f68a-4672-b266-f3562154baa3\" (UID: \"5eb2c17f-f68a-4672-b266-f3562154baa3\") "
Oct 30 01:20:31.121958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.121931    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a003430b-13be-47b6-864c-9062ae70f676-plugins-config-volume" pod "a003430b-13be-47b6-864c-9062ae70f676" (UID: "a003430b-13be-47b6-864c-9062ae70f676"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:31.122432 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.122396    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb2c17f-f68a-4672-b266-f3562154baa3-plugins-config-volume" pod "5eb2c17f-f68a-4672-b266-f3562154baa3" (UID: "5eb2c17f-f68a-4672-b266-f3562154baa3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:31.123466 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.123213    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb2c17f-f68a-4672-b266-f3562154baa3-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:31.123466 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.123243    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a003430b-13be-47b6-864c-9062ae70f676-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:31.128748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.128458    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb2c17f-f68a-4672-b266-f3562154baa3-kube-api-access-lbdkj" pod "5eb2c17f-f68a-4672-b266-f3562154baa3" (UID: "5eb2c17f-f68a-4672-b266-f3562154baa3"). InnerVolumeSpecName "kube-api-access-lbdkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:31.130961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.130928    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a003430b-13be-47b6-864c-9062ae70f676-kube-api-access-nmf4b" pod "a003430b-13be-47b6-864c-9062ae70f676" (UID: "a003430b-13be-47b6-864c-9062ae70f676"). InnerVolumeSpecName "kube-api-access-nmf4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:31.138203 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.138175    6099 generic.go:356] "Generic (PLEG): container finished" podID="a003430b-13be-47b6-864c-9062ae70f676" containerID="307aba3e165a282d1d9ec24ecb2f268308dea98948b0cf9e8058c95792938335" exitCode=0
Oct 30 01:20:31.138314 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.138273    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx"
Oct 30 01:20:31.138387 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.138309    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx" event={"ID":"a003430b-13be-47b6-864c-9062ae70f676","Type":"ContainerDied","Data":"307aba3e165a282d1d9ec24ecb2f268308dea98948b0cf9e8058c95792938335"}
Oct 30 01:20:31.138453 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.138430    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx" event={"ID":"a003430b-13be-47b6-864c-9062ae70f676","Type":"ContainerDied","Data":"f9af8eae4a71cbd8d60fa34fca88ce3286bc12d180010a772f3b11d676812bb4"}
Oct 30 01:20:31.138513 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.138457    6099 scope.go:122] "RemoveContainer" containerID="307aba3e165a282d1d9ec24ecb2f268308dea98948b0cf9e8058c95792938335"
Oct 30 01:20:31.150220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.150189    6099 generic.go:356] "Generic (PLEG): container finished" podID="5eb2c17f-f68a-4672-b266-f3562154baa3" containerID="347b93dfefbcdc70315113b5e4bc91862a2a1406f877c10398a06a42761a823f" exitCode=0
Oct 30 01:20:31.150356 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.150251    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s" event={"ID":"5eb2c17f-f68a-4672-b266-f3562154baa3","Type":"ContainerDied","Data":"347b93dfefbcdc70315113b5e4bc91862a2a1406f877c10398a06a42761a823f"}
Oct 30 01:20:31.150356 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.150309    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s" event={"ID":"5eb2c17f-f68a-4672-b266-f3562154baa3","Type":"ContainerDied","Data":"d01a035737fb1873d3275efe7389f95ee54dc6bd53ca2b04c4686f70b57c6c66"}
Oct 30 01:20:31.150463 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.150382    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s"
Oct 30 01:20:31.164570 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.164544    6099 scope.go:122] "RemoveContainer" containerID="307aba3e165a282d1d9ec24ecb2f268308dea98948b0cf9e8058c95792938335"
Oct 30 01:20:31.166057 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:20:31.165940    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"307aba3e165a282d1d9ec24ecb2f268308dea98948b0cf9e8058c95792938335\": container with ID starting with 307aba3e165a282d1d9ec24ecb2f268308dea98948b0cf9e8058c95792938335 not found: ID does not exist" containerID="307aba3e165a282d1d9ec24ecb2f268308dea98948b0cf9e8058c95792938335"
Oct 30 01:20:31.166294 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.166148    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"307aba3e165a282d1d9ec24ecb2f268308dea98948b0cf9e8058c95792938335"} err="failed to get container status \"307aba3e165a282d1d9ec24ecb2f268308dea98948b0cf9e8058c95792938335\": rpc error: code = NotFound desc = could not find container \"307aba3e165a282d1d9ec24ecb2f268308dea98948b0cf9e8058c95792938335\": container with ID starting with 307aba3e165a282d1d9ec24ecb2f268308dea98948b0cf9e8058c95792938335 not found: ID does not exist"
Oct 30 01:20:31.166503 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.166301    6099 scope.go:122] "RemoveContainer" containerID="347b93dfefbcdc70315113b5e4bc91862a2a1406f877c10398a06a42761a823f"
Oct 30 01:20:31.180956 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.180731    6099 scope.go:122] "RemoveContainer" containerID="347b93dfefbcdc70315113b5e4bc91862a2a1406f877c10398a06a42761a823f"
Oct 30 01:20:31.181083 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:20:31.181034    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"347b93dfefbcdc70315113b5e4bc91862a2a1406f877c10398a06a42761a823f\": container with ID starting with 347b93dfefbcdc70315113b5e4bc91862a2a1406f877c10398a06a42761a823f not found: ID does not exist" containerID="347b93dfefbcdc70315113b5e4bc91862a2a1406f877c10398a06a42761a823f"
Oct 30 01:20:31.181083 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.181067    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"347b93dfefbcdc70315113b5e4bc91862a2a1406f877c10398a06a42761a823f"} err="failed to get container status \"347b93dfefbcdc70315113b5e4bc91862a2a1406f877c10398a06a42761a823f\": rpc error: code = NotFound desc = could not find container \"347b93dfefbcdc70315113b5e4bc91862a2a1406f877c10398a06a42761a823f\": container with ID starting with 347b93dfefbcdc70315113b5e4bc91862a2a1406f877c10398a06a42761a823f not found: ID does not exist"
Oct 30 01:20:31.209748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.208992    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx"]
Oct 30 01:20:31.215230 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.215203    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6c4f6f-69b667986-p2bwx"]
Oct 30 01:20:31.224932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.224905    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbdkj\" (UniqueName: \"kubernetes.io/projected/5eb2c17f-f68a-4672-b266-f3562154baa3-kube-api-access-lbdkj\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:31.225064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.224934    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmf4b\" (UniqueName: \"kubernetes.io/projected/a003430b-13be-47b6-864c-9062ae70f676-kube-api-access-nmf4b\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:31.243682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.243631    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s"]
Oct 30 01:20:31.247502 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:31.247474    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c4f6f-755bc8fb4c-jhs9s"]
Oct 30 01:20:32.311380 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:32.311347    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb2c17f-f68a-4672-b266-f3562154baa3" path="/var/lib/kubelet/pods/5eb2c17f-f68a-4672-b266-f3562154baa3/volumes"
Oct 30 01:20:32.312098 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:32.312077    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a003430b-13be-47b6-864c-9062ae70f676" path="/var/lib/kubelet/pods/a003430b-13be-47b6-864c-9062ae70f676/volumes"
Oct 30 01:20:33.279248 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:33.279211    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr"]
Oct 30 01:20:38.279792 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.279755    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr" podUID="a627558d-fddd-4135-be1e-b24b424d0479" containerName="payload-processing" containerID="cri-o://6dc1574144ff464579fc22278ad6e3659e0b9e64e4953dfa3d6d4cd3dc4ee23e" gracePeriod=25
Oct 30 01:20:38.579603 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.579570    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main" containerID="cri-o://7392df5c2aa3bc70f467cc7629cde4fa7dd0dea3ae6c8be8e1428d7ee1b0c202" gracePeriod=45
Oct 30 01:20:38.582946 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.582922    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr"
Oct 30 01:20:38.664467 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.664434    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a627558d-fddd-4135-be1e-b24b424d0479-kube-api-access-x5792\" (UniqueName: \"kubernetes.io/projected/a627558d-fddd-4135-be1e-b24b424d0479-kube-api-access-x5792\") pod \"a627558d-fddd-4135-be1e-b24b424d0479\" (UID: \"a627558d-fddd-4135-be1e-b24b424d0479\") "
Oct 30 01:20:38.664644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.664556    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a627558d-fddd-4135-be1e-b24b424d0479-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a627558d-fddd-4135-be1e-b24b424d0479-plugins-config-volume\") pod \"a627558d-fddd-4135-be1e-b24b424d0479\" (UID: \"a627558d-fddd-4135-be1e-b24b424d0479\") "
Oct 30 01:20:38.664943 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.664918    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a627558d-fddd-4135-be1e-b24b424d0479-plugins-config-volume" pod "a627558d-fddd-4135-be1e-b24b424d0479" (UID: "a627558d-fddd-4135-be1e-b24b424d0479"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:38.665179 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.665162    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a627558d-fddd-4135-be1e-b24b424d0479-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:38.666619 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.666594    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a627558d-fddd-4135-be1e-b24b424d0479-kube-api-access-x5792" pod "a627558d-fddd-4135-be1e-b24b424d0479" (UID: "a627558d-fddd-4135-be1e-b24b424d0479"). InnerVolumeSpecName "kube-api-access-x5792". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:38.768740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.767065    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5792\" (UniqueName: \"kubernetes.io/projected/a627558d-fddd-4135-be1e-b24b424d0479-kube-api-access-x5792\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:38.857970 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.857941    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 01:20:38.976694 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.976664    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-home\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-home\") pod \"93ef078d-6084-421b-9311-c91a4852be8b\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") "
Oct 30 01:20:38.976896 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.976732    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93ef078d-6084-421b-9311-c91a4852be8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/93ef078d-6084-421b-9311-c91a4852be8b-tls-certs\") pod \"93ef078d-6084-421b-9311-c91a4852be8b\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") "
Oct 30 01:20:38.976896 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.976776    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-tmp-dir\") pod \"93ef078d-6084-421b-9311-c91a4852be8b\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") "
Oct 30 01:20:38.977041 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.977004    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-dshm\") pod \"93ef078d-6084-421b-9311-c91a4852be8b\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") "
Oct 30 01:20:38.977143 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.977106    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-model-cache\") pod \"93ef078d-6084-421b-9311-c91a4852be8b\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") "
Oct 30 01:20:38.977143 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.976988    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-home" pod "93ef078d-6084-421b-9311-c91a4852be8b" (UID: "93ef078d-6084-421b-9311-c91a4852be8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:38.977278 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.977144    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-tmp-dir" pod "93ef078d-6084-421b-9311-c91a4852be8b" (UID: "93ef078d-6084-421b-9311-c91a4852be8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:38.977278 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.977227    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ef078d-6084-421b-9311-c91a4852be8b-kube-api-access-dgnfk\" (UniqueName: \"kubernetes.io/projected/93ef078d-6084-421b-9311-c91a4852be8b-kube-api-access-dgnfk\") pod \"93ef078d-6084-421b-9311-c91a4852be8b\" (UID: \"93ef078d-6084-421b-9311-c91a4852be8b\") "
Oct 30 01:20:38.977384 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.977367    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-model-cache" pod "93ef078d-6084-421b-9311-c91a4852be8b" (UID: "93ef078d-6084-421b-9311-c91a4852be8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:38.978123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.978097    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:38.978247 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.978126    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:38.978247 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.978141    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:38.979187 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.979159    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-dshm" pod "93ef078d-6084-421b-9311-c91a4852be8b" (UID: "93ef078d-6084-421b-9311-c91a4852be8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:38.979415 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.979390    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93ef078d-6084-421b-9311-c91a4852be8b-tls-certs" pod "93ef078d-6084-421b-9311-c91a4852be8b" (UID: "93ef078d-6084-421b-9311-c91a4852be8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:38.979494 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:38.979480    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ef078d-6084-421b-9311-c91a4852be8b-kube-api-access-dgnfk" pod "93ef078d-6084-421b-9311-c91a4852be8b" (UID: "93ef078d-6084-421b-9311-c91a4852be8b"). InnerVolumeSpecName "kube-api-access-dgnfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:39.079774 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.079740    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ef078d-6084-421b-9311-c91a4852be8b-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:39.079774 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.079765    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgnfk\" (UniqueName: \"kubernetes.io/projected/93ef078d-6084-421b-9311-c91a4852be8b-kube-api-access-dgnfk\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:39.079774 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.079776    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93ef078d-6084-421b-9311-c91a4852be8b-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:39.224673 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.224582    6099 generic.go:356] "Generic (PLEG): container finished" podID="a627558d-fddd-4135-be1e-b24b424d0479" containerID="6dc1574144ff464579fc22278ad6e3659e0b9e64e4953dfa3d6d4cd3dc4ee23e" exitCode=0
Oct 30 01:20:39.224986 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.224638    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr"
Oct 30 01:20:39.225108 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.224656    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr" event={"ID":"a627558d-fddd-4135-be1e-b24b424d0479","Type":"ContainerDied","Data":"6dc1574144ff464579fc22278ad6e3659e0b9e64e4953dfa3d6d4cd3dc4ee23e"}
Oct 30 01:20:39.225169 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.225150    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr" event={"ID":"a627558d-fddd-4135-be1e-b24b424d0479","Type":"ContainerDied","Data":"2f2f6d93e8aff45b1e689ebf9cc8210f0a271beed17e334a749bb2522d85d618"}
Oct 30 01:20:39.225234 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.225218    6099 scope.go:122] "RemoveContainer" containerID="6dc1574144ff464579fc22278ad6e3659e0b9e64e4953dfa3d6d4cd3dc4ee23e"
Oct 30 01:20:39.227399 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.227371    6099 generic.go:356] "Generic (PLEG): container finished" podID="93ef078d-6084-421b-9311-c91a4852be8b" containerID="7392df5c2aa3bc70f467cc7629cde4fa7dd0dea3ae6c8be8e1428d7ee1b0c202" exitCode=0
Oct 30 01:20:39.227572 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.227555    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" event={"ID":"93ef078d-6084-421b-9311-c91a4852be8b","Type":"ContainerDied","Data":"7392df5c2aa3bc70f467cc7629cde4fa7dd0dea3ae6c8be8e1428d7ee1b0c202"}
Oct 30 01:20:39.227645 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.227587    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk" event={"ID":"93ef078d-6084-421b-9311-c91a4852be8b","Type":"ContainerDied","Data":"6d8a204d2dc946a5040408503e911faf0f3538a66dff6422ff2dd00f7e914b61"}
Oct 30 01:20:39.227760 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.227651    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"
Oct 30 01:20:39.243311 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.243293    6099 scope.go:122] "RemoveContainer" containerID="6dc1574144ff464579fc22278ad6e3659e0b9e64e4953dfa3d6d4cd3dc4ee23e"
Oct 30 01:20:39.243581 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:20:39.243562    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc1574144ff464579fc22278ad6e3659e0b9e64e4953dfa3d6d4cd3dc4ee23e\": container with ID starting with 6dc1574144ff464579fc22278ad6e3659e0b9e64e4953dfa3d6d4cd3dc4ee23e not found: ID does not exist" containerID="6dc1574144ff464579fc22278ad6e3659e0b9e64e4953dfa3d6d4cd3dc4ee23e"
Oct 30 01:20:39.243641 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.243588    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc1574144ff464579fc22278ad6e3659e0b9e64e4953dfa3d6d4cd3dc4ee23e"} err="failed to get container status \"6dc1574144ff464579fc22278ad6e3659e0b9e64e4953dfa3d6d4cd3dc4ee23e\": rpc error: code = NotFound desc = could not find container \"6dc1574144ff464579fc22278ad6e3659e0b9e64e4953dfa3d6d4cd3dc4ee23e\": container with ID starting with 6dc1574144ff464579fc22278ad6e3659e0b9e64e4953dfa3d6d4cd3dc4ee23e not found: ID does not exist"
Oct 30 01:20:39.243641 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.243603    6099 scope.go:122] "RemoveContainer" containerID="7392df5c2aa3bc70f467cc7629cde4fa7dd0dea3ae6c8be8e1428d7ee1b0c202"
Oct 30 01:20:39.258569 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.258548    6099 scope.go:122] "RemoveContainer" containerID="7392df5c2aa3bc70f467cc7629cde4fa7dd0dea3ae6c8be8e1428d7ee1b0c202"
Oct 30 01:20:39.258855 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:20:39.258835    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7392df5c2aa3bc70f467cc7629cde4fa7dd0dea3ae6c8be8e1428d7ee1b0c202\": container with ID starting with 7392df5c2aa3bc70f467cc7629cde4fa7dd0dea3ae6c8be8e1428d7ee1b0c202 not found: ID does not exist" containerID="7392df5c2aa3bc70f467cc7629cde4fa7dd0dea3ae6c8be8e1428d7ee1b0c202"
Oct 30 01:20:39.258901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.258862    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7392df5c2aa3bc70f467cc7629cde4fa7dd0dea3ae6c8be8e1428d7ee1b0c202"} err="failed to get container status \"7392df5c2aa3bc70f467cc7629cde4fa7dd0dea3ae6c8be8e1428d7ee1b0c202\": rpc error: code = NotFound desc = could not find container \"7392df5c2aa3bc70f467cc7629cde4fa7dd0dea3ae6c8be8e1428d7ee1b0c202\": container with ID starting with 7392df5c2aa3bc70f467cc7629cde4fa7dd0dea3ae6c8be8e1428d7ee1b0c202 not found: ID does not exist"
Oct 30 01:20:39.281452 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.281428    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"]
Oct 30 01:20:39.293124 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.293100    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b3593a/simulator-w3-b3593a-kserve-74b95c566f-bb5sk"]
Oct 30 01:20:39.330351 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.330316    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr"]
Oct 30 01:20:39.336545 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:39.336513    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a42e5-8b4998c78-p9lfr"]
Oct 30 01:20:40.265009 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:40.264972    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93ef078d-6084-421b-9311-c91a4852be8b" path="/var/lib/kubelet/pods/93ef078d-6084-421b-9311-c91a4852be8b/volumes"
Oct 30 01:20:40.265559 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:40.265544    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a627558d-fddd-4135-be1e-b24b424d0479" path="/var/lib/kubelet/pods/a627558d-fddd-4135-be1e-b24b424d0479/volumes"
Oct 30 01:20:45.529663 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:45.529609    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"]
Oct 30 01:20:45.530594 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:45.529836    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f" podUID="1c5c9922-e50a-47a1-8b74-021582765920" containerName="istio-proxy" containerID="cri-o://b09e931d70350390da13f5134a9ea4c65a6798eb98466bf4173cd5e5e80facd7" gracePeriod=30
Oct 30 01:20:46.631251 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:20:46.631217    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:20:46.992486 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:46.992402    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"]
Oct 30 01:20:46.992675 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:46.992653    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps" podUID="48ab3ee2-9839-4203-a173-c6841b95d45a" containerName="maas-api" containerID="cri-o://72e26aa1957e76249b612598e2d297757212fc198ac249d2bdccde5d17ae0eb1" gracePeriod=30
Oct 30 01:20:47.680163 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:47.680063    6099 generic.go:356] "Generic (PLEG): container finished" podID="48ab3ee2-9839-4203-a173-c6841b95d45a" containerID="72e26aa1957e76249b612598e2d297757212fc198ac249d2bdccde5d17ae0eb1" exitCode=0
Oct 30 01:20:47.681106 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:47.681072    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps" event={"ID":"48ab3ee2-9839-4203-a173-c6841b95d45a","Type":"ContainerDied","Data":"72e26aa1957e76249b612598e2d297757212fc198ac249d2bdccde5d17ae0eb1"}
Oct 30 01:20:47.683238 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:47.683217    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:20:47.788493 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:47.788455    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48ab3ee2-9839-4203-a173-c6841b95d45a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48ab3ee2-9839-4203-a173-c6841b95d45a-maas-api-tls\") pod \"48ab3ee2-9839-4203-a173-c6841b95d45a\" (UID: \"48ab3ee2-9839-4203-a173-c6841b95d45a\") "
Oct 30 01:20:47.788667 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:47.788527    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48ab3ee2-9839-4203-a173-c6841b95d45a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/48ab3ee2-9839-4203-a173-c6841b95d45a-metrics-certs\") pod \"48ab3ee2-9839-4203-a173-c6841b95d45a\" (UID: \"48ab3ee2-9839-4203-a173-c6841b95d45a\") "
Oct 30 01:20:47.788667 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:47.788613    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48ab3ee2-9839-4203-a173-c6841b95d45a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ab3ee2-9839-4203-a173-c6841b95d45a-openshift-service-ca\") pod \"48ab3ee2-9839-4203-a173-c6841b95d45a\" (UID: \"48ab3ee2-9839-4203-a173-c6841b95d45a\") "
Oct 30 01:20:47.788991 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:47.788968    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ab3ee2-9839-4203-a173-c6841b95d45a-openshift-service-ca" pod "48ab3ee2-9839-4203-a173-c6841b95d45a" (UID: "48ab3ee2-9839-4203-a173-c6841b95d45a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:47.789121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:47.789105    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ab3ee2-9839-4203-a173-c6841b95d45a-kube-api-access-575qb\" (UniqueName: \"kubernetes.io/projected/48ab3ee2-9839-4203-a173-c6841b95d45a-kube-api-access-575qb\") pod \"48ab3ee2-9839-4203-a173-c6841b95d45a\" (UID: \"48ab3ee2-9839-4203-a173-c6841b95d45a\") "
Oct 30 01:20:47.790097 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:47.790071    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ab3ee2-9839-4203-a173-c6841b95d45a-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:47.791649 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:47.791620    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48ab3ee2-9839-4203-a173-c6841b95d45a-metrics-certs" pod "48ab3ee2-9839-4203-a173-c6841b95d45a" (UID: "48ab3ee2-9839-4203-a173-c6841b95d45a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:47.793489 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:47.793462    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48ab3ee2-9839-4203-a173-c6841b95d45a-maas-api-tls" pod "48ab3ee2-9839-4203-a173-c6841b95d45a" (UID: "48ab3ee2-9839-4203-a173-c6841b95d45a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:47.796415 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:47.796074    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ab3ee2-9839-4203-a173-c6841b95d45a-kube-api-access-575qb" pod "48ab3ee2-9839-4203-a173-c6841b95d45a" (UID: "48ab3ee2-9839-4203-a173-c6841b95d45a"). InnerVolumeSpecName "kube-api-access-575qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:47.891991 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:47.891950    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48ab3ee2-9839-4203-a173-c6841b95d45a-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:47.891991 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:47.891989    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48ab3ee2-9839-4203-a173-c6841b95d45a-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:47.892253 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:47.892005    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-575qb\" (UniqueName: \"kubernetes.io/projected/48ab3ee2-9839-4203-a173-c6841b95d45a-kube-api-access-575qb\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:48.335873 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:48.335842    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rcxc5_1569a420-ef8b-4ca9-86c1-cde78845b4b6/payload-processing/0.log"
Oct 30 01:20:48.527025 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:48.526945    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rcxc5_1569a420-ef8b-4ca9-86c1-cde78845b4b6/payload-processing/0.log"
Oct 30 01:20:48.690077 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:48.690045    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"
Oct 30 01:20:48.690527 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:48.690075    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps" event={"ID":"48ab3ee2-9839-4203-a173-c6841b95d45a","Type":"ContainerDied","Data":"61d757040a761a9c5979e21b809f92937370cb1e204fb5b93f4c3d7bc23fd307"}
Oct 30 01:20:48.690527 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:48.690133    6099 scope.go:122] "RemoveContainer" containerID="72e26aa1957e76249b612598e2d297757212fc198ac249d2bdccde5d17ae0eb1"
Oct 30 01:20:48.773928 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:48.773900    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"]
Oct 30 01:20:48.786763 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:48.786693    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c4f6f-f79f4d764-gbcps"]
Oct 30 01:20:49.343148 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.343119    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59"]
Oct 30 01:20:49.343908 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.343893    6099 cpu_manager.go:414] "containerMap: removing container" podUID="f0bbb319-2301-41ef-879f-29ea51d8699e" containerName="istio-proxy"
Oct 30 01:20:49.343962 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.343908    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0bbb319-2301-41ef-879f-29ea51d8699e" containerName="istio-proxy"
Oct 30 01:20:49.343962 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.343920    6099 cpu_manager.go:414] "containerMap: removing container" podUID="48ab3ee2-9839-4203-a173-c6841b95d45a" containerName="maas-api"
Oct 30 01:20:49.343962 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.343926    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ab3ee2-9839-4203-a173-c6841b95d45a" containerName="maas-api"
Oct 30 01:20:49.343962 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.343941    6099 cpu_manager.go:414] "containerMap: removing container" podUID="a003430b-13be-47b6-864c-9062ae70f676" containerName="payload-processing"
Oct 30 01:20:49.343962 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.343946    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a003430b-13be-47b6-864c-9062ae70f676" containerName="payload-processing"
Oct 30 01:20:49.343962 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.343954    6099 cpu_manager.go:414] "containerMap: removing container" podUID="7fb31e16-f069-4549-ab6b-c8cdb62b43aa" containerName="maas-api"
Oct 30 01:20:49.343962 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.343959    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb31e16-f069-4549-ab6b-c8cdb62b43aa" containerName="maas-api"
Oct 30 01:20:49.344171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.343976    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e2c83697-c779-4359-804d-b6b4b4d5e77c" containerName="istio-proxy"
Oct 30 01:20:49.344171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.343981    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2c83697-c779-4359-804d-b6b4b4d5e77c" containerName="istio-proxy"
Oct 30 01:20:49.344171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.343992    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5eb2c17f-f68a-4672-b266-f3562154baa3" containerName="payload-pre-processing"
Oct 30 01:20:49.344171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.343997    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb2c17f-f68a-4672-b266-f3562154baa3" containerName="payload-pre-processing"
Oct 30 01:20:49.344171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344016    6099 cpu_manager.go:414] "containerMap: removing container" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main"
Oct 30 01:20:49.344171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344021    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main"
Oct 30 01:20:49.344171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344032    6099 cpu_manager.go:414] "containerMap: removing container" podUID="d8655eb2-c236-49f0-90e8-b7b4ee6e51be" containerName="istio-proxy"
Oct 30 01:20:49.344171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344037    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8655eb2-c236-49f0-90e8-b7b4ee6e51be" containerName="istio-proxy"
Oct 30 01:20:49.344171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344044    6099 cpu_manager.go:414] "containerMap: removing container" podUID="a627558d-fddd-4135-be1e-b24b424d0479" containerName="payload-processing"
Oct 30 01:20:49.344171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344049    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a627558d-fddd-4135-be1e-b24b424d0479" containerName="payload-processing"
Oct 30 01:20:49.344171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344062    6099 cpu_manager.go:414] "containerMap: removing container" podUID="d8655eb2-c236-49f0-90e8-b7b4ee6e51be" containerName="istio-proxy"
Oct 30 01:20:49.344171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344067    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8655eb2-c236-49f0-90e8-b7b4ee6e51be" containerName="istio-proxy"
Oct 30 01:20:49.344171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344165    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0bbb319-2301-41ef-879f-29ea51d8699e" containerName="istio-proxy"
Oct 30 01:20:49.344171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344173    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ab3ee2-9839-4203-a173-c6841b95d45a" containerName="maas-api"
Oct 30 01:20:49.344682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344183    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a003430b-13be-47b6-864c-9062ae70f676" containerName="payload-processing"
Oct 30 01:20:49.344682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344190    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb31e16-f069-4549-ab6b-c8cdb62b43aa" containerName="maas-api"
Oct 30 01:20:49.344682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344197    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2c83697-c779-4359-804d-b6b4b4d5e77c" containerName="istio-proxy"
Oct 30 01:20:49.344682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344204    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb2c17f-f68a-4672-b266-f3562154baa3" containerName="payload-pre-processing"
Oct 30 01:20:49.344682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344209    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2c83697-c779-4359-804d-b6b4b4d5e77c" containerName="istio-proxy"
Oct 30 01:20:49.344682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344216    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ef078d-6084-421b-9311-c91a4852be8b" containerName="main"
Oct 30 01:20:49.344682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344222    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8655eb2-c236-49f0-90e8-b7b4ee6e51be" containerName="istio-proxy"
Oct 30 01:20:49.344682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344227    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a627558d-fddd-4135-be1e-b24b424d0479" containerName="payload-processing"
Oct 30 01:20:49.344682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.344235    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8655eb2-c236-49f0-90e8-b7b4ee6e51be" containerName="istio-proxy"
Oct 30 01:20:49.349040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.349020    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59"
Oct 30 01:20:49.370276 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.370252    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59"]
Oct 30 01:20:49.391250 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.391225    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:20:49.391409 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.391298    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:20:49.422690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.422654    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz8bw\" (UniqueName: \"kubernetes.io/projected/41fd810f-4559-492b-8038-7cd547121618-kube-api-access-xz8bw\") pod \"maas-api-revoke-keys-e2e-ipp-a-481249-dnj59\" (UID: \"41fd810f-4559-492b-8038-7cd547121618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59"
Oct 30 01:20:49.422873 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.422822    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41fd810f-4559-492b-8038-7cd547121618-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-481249-dnj59\" (UID: \"41fd810f-4559-492b-8038-7cd547121618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59"
Oct 30 01:20:49.422979 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.422892    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41fd810f-4559-492b-8038-7cd547121618-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-481249-dnj59\" (UID: \"41fd810f-4559-492b-8038-7cd547121618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59"
Oct 30 01:20:49.524084 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.524052    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41fd810f-4559-492b-8038-7cd547121618-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-481249-dnj59\" (UID: \"41fd810f-4559-492b-8038-7cd547121618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59"
Oct 30 01:20:49.524256 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.524146    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41fd810f-4559-492b-8038-7cd547121618-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-481249-dnj59\" (UID: \"41fd810f-4559-492b-8038-7cd547121618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59"
Oct 30 01:20:49.524315 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.524303    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz8bw\" (UniqueName: \"kubernetes.io/projected/41fd810f-4559-492b-8038-7cd547121618-kube-api-access-xz8bw\") pod \"maas-api-revoke-keys-e2e-ipp-a-481249-dnj59\" (UID: \"41fd810f-4559-492b-8038-7cd547121618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59"
Oct 30 01:20:49.524870 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.524848    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41fd810f-4559-492b-8038-7cd547121618-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-481249-dnj59\" (UID: \"41fd810f-4559-492b-8038-7cd547121618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59"
Oct 30 01:20:49.526591 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.526564    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41fd810f-4559-492b-8038-7cd547121618-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-481249-dnj59\" (UID: \"41fd810f-4559-492b-8038-7cd547121618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59"
Oct 30 01:20:49.547627 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.547599    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz8bw\" (UniqueName: \"kubernetes.io/projected/41fd810f-4559-492b-8038-7cd547121618-kube-api-access-xz8bw\") pod \"maas-api-revoke-keys-e2e-ipp-a-481249-dnj59\" (UID: \"41fd810f-4559-492b-8038-7cd547121618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59"
Oct 30 01:20:49.665509 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.665430    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59"
Oct 30 01:20:49.984964 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:49.984933    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59"]
Oct 30 01:20:49.985908 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:20:49.985882    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd810f_4559_492b_8038_7cd547121618.slice/crio-f7df75b151b7b64b77cad2acb43c489d1913855af681a35b1cbfd820c4c03730 WatchSource:0}: Error finding container f7df75b151b7b64b77cad2acb43c489d1913855af681a35b1cbfd820c4c03730: Status 404 returned error can't find the container with id f7df75b151b7b64b77cad2acb43c489d1913855af681a35b1cbfd820c4c03730
Oct 30 01:20:50.330675 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.329058    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ab3ee2-9839-4203-a173-c6841b95d45a" path="/var/lib/kubelet/pods/48ab3ee2-9839-4203-a173-c6841b95d45a/volumes"
Oct 30 01:20:50.710668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.710630    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59" event={"ID":"41fd810f-4559-492b-8038-7cd547121618","Type":"ContainerStarted","Data":"6f208a5ac28415e9799eee4e1731c978f732e8dcc34442bfe6f4109a7361e627"}
Oct 30 01:20:50.710668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.710692    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59" event={"ID":"41fd810f-4559-492b-8038-7cd547121618","Type":"ContainerStarted","Data":"f7df75b151b7b64b77cad2acb43c489d1913855af681a35b1cbfd820c4c03730"}
Oct 30 01:20:50.713849 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.713078    6099 generic.go:356] "Generic (PLEG): container finished" podID="1c5c9922-e50a-47a1-8b74-021582765920" containerID="b09e931d70350390da13f5134a9ea4c65a6798eb98466bf4173cd5e5e80facd7" exitCode=0
Oct 30 01:20:50.713849 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.713173    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f" event={"ID":"1c5c9922-e50a-47a1-8b74-021582765920","Type":"ContainerDied","Data":"b09e931d70350390da13f5134a9ea4c65a6798eb98466bf4173cd5e5e80facd7"}
Oct 30 01:20:50.743104 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.743067    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7"]
Oct 30 01:20:50.744444 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.744414    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e2c83697-c779-4359-804d-b6b4b4d5e77c" containerName="istio-proxy"
Oct 30 01:20:50.744444 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.744443    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2c83697-c779-4359-804d-b6b4b4d5e77c" containerName="istio-proxy"
Oct 30 01:20:50.751297 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.751245    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59" podStartSLOduration=1.7512296680000001 podStartE2EDuration="1.751229668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:50.750257617 +0000 UTC m=+2744.607165372" watchObservedRunningTime="2026-10-30 01:20:50.751229668 +0000 UTC m=+2744.608137421"
Oct 30 01:20:50.754205 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.754184    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7"
Oct 30 01:20:50.760908 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.760873    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7"]
Oct 30 01:20:50.849593 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.849563    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:20:50.859887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.859856    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjxh\" (UniqueName: \"kubernetes.io/projected/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-kube-api-access-7kjxh\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7\" (UID: \"1fc8dc61-d744-4c4b-89c5-4122ca0edd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7"
Oct 30 01:20:50.859993 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.859908    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7\" (UID: \"1fc8dc61-d744-4c4b-89c5-4122ca0edd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7"
Oct 30 01:20:50.860032 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.859986    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7\" (UID: \"1fc8dc61-d744-4c4b-89c5-4122ca0edd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7"
Oct 30 01:20:50.961167 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.961075    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c5c9922-e50a-47a1-8b74-021582765920-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c5c9922-e50a-47a1-8b74-021582765920-istio-podinfo\") pod \"1c5c9922-e50a-47a1-8b74-021582765920\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") "
Oct 30 01:20:50.961167 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.961136    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-istio-data\") pod \"1c5c9922-e50a-47a1-8b74-021582765920\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") "
Oct 30 01:20:50.961407 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.961302    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-workload-socket\") pod \"1c5c9922-e50a-47a1-8b74-021582765920\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") "
Oct 30 01:20:50.961521 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.961495    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-istio-data" pod "1c5c9922-e50a-47a1-8b74-021582765920" (UID: "1c5c9922-e50a-47a1-8b74-021582765920"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:50.961605 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.961517    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-workload-socket" pod "1c5c9922-e50a-47a1-8b74-021582765920" (UID: "1c5c9922-e50a-47a1-8b74-021582765920"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:50.961605 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.961590    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c5c9922-e50a-47a1-8b74-021582765920-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c5c9922-e50a-47a1-8b74-021582765920-istiod-ca-cert\") pod \"1c5c9922-e50a-47a1-8b74-021582765920\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") "
Oct 30 01:20:50.961753 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.961683    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-istio-envoy\") pod \"1c5c9922-e50a-47a1-8b74-021582765920\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") "
Oct 30 01:20:50.961753 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.961742    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5c9922-e50a-47a1-8b74-021582765920-kube-api-access-f6564\" (UniqueName: \"kubernetes.io/projected/1c5c9922-e50a-47a1-8b74-021582765920-kube-api-access-f6564\") pod \"1c5c9922-e50a-47a1-8b74-021582765920\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") "
Oct 30 01:20:50.961884 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.961868    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-workload-certs\") pod \"1c5c9922-e50a-47a1-8b74-021582765920\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") "
Oct 30 01:20:50.961964 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.961948    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-credential-socket\") pod \"1c5c9922-e50a-47a1-8b74-021582765920\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") "
Oct 30 01:20:50.962035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.961997    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5c9922-e50a-47a1-8b74-021582765920-istiod-ca-cert" pod "1c5c9922-e50a-47a1-8b74-021582765920" (UID: "1c5c9922-e50a-47a1-8b74-021582765920"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:50.962035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.962019    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5c9922-e50a-47a1-8b74-021582765920-istio-token\" (UniqueName: \"kubernetes.io/projected/1c5c9922-e50a-47a1-8b74-021582765920-istio-token\") pod \"1c5c9922-e50a-47a1-8b74-021582765920\" (UID: \"1c5c9922-e50a-47a1-8b74-021582765920\") "
Oct 30 01:20:50.962294 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.962272    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-workload-certs" pod "1c5c9922-e50a-47a1-8b74-021582765920" (UID: "1c5c9922-e50a-47a1-8b74-021582765920"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:50.962436 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.962415    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjxh\" (UniqueName: \"kubernetes.io/projected/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-kube-api-access-7kjxh\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7\" (UID: \"1fc8dc61-d744-4c4b-89c5-4122ca0edd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7"
Oct 30 01:20:50.962511 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.962493    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7\" (UID: \"1fc8dc61-d744-4c4b-89c5-4122ca0edd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7"
Oct 30 01:20:50.962568 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.962546    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7\" (UID: \"1fc8dc61-d744-4c4b-89c5-4122ca0edd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7"
Oct 30 01:20:50.963035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.963008    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:50.963140 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.963040    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:50.963140 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.963058    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c5c9922-e50a-47a1-8b74-021582765920-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:50.963140 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.963074    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:50.963472 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.963419    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7\" (UID: \"1fc8dc61-d744-4c4b-89c5-4122ca0edd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7"
Oct 30 01:20:50.963835 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.963809    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-credential-socket" pod "1c5c9922-e50a-47a1-8b74-021582765920" (UID: "1c5c9922-e50a-47a1-8b74-021582765920"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:50.965178 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.965140    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c5c9922-e50a-47a1-8b74-021582765920-istio-podinfo" pod "1c5c9922-e50a-47a1-8b74-021582765920" (UID: "1c5c9922-e50a-47a1-8b74-021582765920"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:20:50.965874 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.965845    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5c9922-e50a-47a1-8b74-021582765920-istio-token" pod "1c5c9922-e50a-47a1-8b74-021582765920" (UID: "1c5c9922-e50a-47a1-8b74-021582765920"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:50.966006 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.965984    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-istio-envoy" pod "1c5c9922-e50a-47a1-8b74-021582765920" (UID: "1c5c9922-e50a-47a1-8b74-021582765920"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:50.966950 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.966925    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5c9922-e50a-47a1-8b74-021582765920-kube-api-access-f6564" pod "1c5c9922-e50a-47a1-8b74-021582765920" (UID: "1c5c9922-e50a-47a1-8b74-021582765920"). InnerVolumeSpecName "kube-api-access-f6564". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:50.967322 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.967305    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7\" (UID: \"1fc8dc61-d744-4c4b-89c5-4122ca0edd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7"
Oct 30 01:20:50.987854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:50.987828    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjxh\" (UniqueName: \"kubernetes.io/projected/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-kube-api-access-7kjxh\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7\" (UID: \"1fc8dc61-d744-4c4b-89c5-4122ca0edd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7"
Oct 30 01:20:51.064416 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:51.064386    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:51.064416 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:51.064414    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6564\" (UniqueName: \"kubernetes.io/projected/1c5c9922-e50a-47a1-8b74-021582765920-kube-api-access-f6564\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:51.064618 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:51.064427    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5c9922-e50a-47a1-8b74-021582765920-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:51.064618 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:51.064436    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c5c9922-e50a-47a1-8b74-021582765920-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:51.064618 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:51.064445    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c5c9922-e50a-47a1-8b74-021582765920-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:51.095616 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:51.094663    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7"
Oct 30 01:20:51.432897 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:20:51.432860    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc8dc61_d744_4c4b_89c5_4122ca0edd22.slice/crio-01634ad4ad41d7d5d6b258bfccfe179db7c305fd2e35c7ad53e05041e3727bb2 WatchSource:0}: Error finding container 01634ad4ad41d7d5d6b258bfccfe179db7c305fd2e35c7ad53e05041e3727bb2: Status 404 returned error can't find the container with id 01634ad4ad41d7d5d6b258bfccfe179db7c305fd2e35c7ad53e05041e3727bb2
Oct 30 01:20:51.433159 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:51.433137    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7"]
Oct 30 01:20:51.722118 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:51.722035    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"
Oct 30 01:20:51.722264 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:51.722039    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f" event={"ID":"1c5c9922-e50a-47a1-8b74-021582765920","Type":"ContainerDied","Data":"33b156e01376bdcaeed0fdf7e6d45bcadb7169d72da64eeb7e664e21e6c8c103"}
Oct 30 01:20:51.722264 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:51.722191    6099 scope.go:122] "RemoveContainer" containerID="b09e931d70350390da13f5134a9ea4c65a6798eb98466bf4173cd5e5e80facd7"
Oct 30 01:20:51.723624 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:51.723599    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7" event={"ID":"1fc8dc61-d744-4c4b-89c5-4122ca0edd22","Type":"ContainerStarted","Data":"01634ad4ad41d7d5d6b258bfccfe179db7c305fd2e35c7ad53e05041e3727bb2"}
Oct 30 01:20:51.725579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:51.725553    6099 generic.go:356] "Generic (PLEG): container finished" podID="41fd810f-4559-492b-8038-7cd547121618" containerID="6f208a5ac28415e9799eee4e1731c978f732e8dcc34442bfe6f4109a7361e627" exitCode=0
Oct 30 01:20:51.725700 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:51.725631    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59" event={"ID":"41fd810f-4559-492b-8038-7cd547121618","Type":"ContainerDied","Data":"6f208a5ac28415e9799eee4e1731c978f732e8dcc34442bfe6f4109a7361e627"}
Oct 30 01:20:51.816927 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:51.816897    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"]
Oct 30 01:20:51.825115 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:51.825087    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-hfl5f"]
Oct 30 01:20:52.264107 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:52.264078    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c5c9922-e50a-47a1-8b74-021582765920" path="/var/lib/kubelet/pods/1c5c9922-e50a-47a1-8b74-021582765920/volumes"
Oct 30 01:20:52.740287 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:52.740253    6099 generic.go:356] "Generic (PLEG): container finished" podID="1fc8dc61-d744-4c4b-89c5-4122ca0edd22" containerID="78997ac4350d149f69ab67bc3078822ab537ac0a5cefb52fb1c182658c9b31d0" exitCode=0
Oct 30 01:20:52.740464 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:52.740334    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7" event={"ID":"1fc8dc61-d744-4c4b-89c5-4122ca0edd22","Type":"ContainerDied","Data":"78997ac4350d149f69ab67bc3078822ab537ac0a5cefb52fb1c182658c9b31d0"}
Oct 30 01:20:52.938620 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:52.938596    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59"
Oct 30 01:20:52.994289 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:52.994217    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41fd810f-4559-492b-8038-7cd547121618-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41fd810f-4559-492b-8038-7cd547121618-maas-api-openshift-ca\") pod \"41fd810f-4559-492b-8038-7cd547121618\" (UID: \"41fd810f-4559-492b-8038-7cd547121618\") "
Oct 30 01:20:52.994442 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:52.994289    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41fd810f-4559-492b-8038-7cd547121618-kube-api-access-xz8bw\" (UniqueName: \"kubernetes.io/projected/41fd810f-4559-492b-8038-7cd547121618-kube-api-access-xz8bw\") pod \"41fd810f-4559-492b-8038-7cd547121618\" (UID: \"41fd810f-4559-492b-8038-7cd547121618\") "
Oct 30 01:20:52.994520 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:52.994499    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41fd810f-4559-492b-8038-7cd547121618-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41fd810f-4559-492b-8038-7cd547121618-maas-api-xks-ca\") pod \"41fd810f-4559-492b-8038-7cd547121618\" (UID: \"41fd810f-4559-492b-8038-7cd547121618\") "
Oct 30 01:20:52.994613 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:52.994588    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41fd810f-4559-492b-8038-7cd547121618-maas-api-openshift-ca" pod "41fd810f-4559-492b-8038-7cd547121618" (UID: "41fd810f-4559-492b-8038-7cd547121618"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:52.995420 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:52.995395    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41fd810f-4559-492b-8038-7cd547121618-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:52.996654 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:52.996627    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fd810f-4559-492b-8038-7cd547121618-kube-api-access-xz8bw" pod "41fd810f-4559-492b-8038-7cd547121618" (UID: "41fd810f-4559-492b-8038-7cd547121618"). InnerVolumeSpecName "kube-api-access-xz8bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:52.997021 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:52.996995    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41fd810f-4559-492b-8038-7cd547121618-maas-api-xks-ca" pod "41fd810f-4559-492b-8038-7cd547121618" (UID: "41fd810f-4559-492b-8038-7cd547121618"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:53.097501 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:53.097466    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz8bw\" (UniqueName: \"kubernetes.io/projected/41fd810f-4559-492b-8038-7cd547121618-kube-api-access-xz8bw\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:53.097501 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:53.097498    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41fd810f-4559-492b-8038-7cd547121618-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:53.749643 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:53.749612    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59"
Oct 30 01:20:53.750098 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:53.749637    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59" event={"ID":"41fd810f-4559-492b-8038-7cd547121618","Type":"ContainerDied","Data":"f7df75b151b7b64b77cad2acb43c489d1913855af681a35b1cbfd820c4c03730"}
Oct 30 01:20:53.750098 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:53.749687    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7df75b151b7b64b77cad2acb43c489d1913855af681a35b1cbfd820c4c03730"
Oct 30 01:20:53.925720 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:53.925685    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7"
Oct 30 01:20:54.015510 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:54.015435    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-maas-api-xks-ca\") pod \"1fc8dc61-d744-4c4b-89c5-4122ca0edd22\" (UID: \"1fc8dc61-d744-4c4b-89c5-4122ca0edd22\") "
Oct 30 01:20:54.015510 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:54.015488    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-maas-api-openshift-ca\") pod \"1fc8dc61-d744-4c4b-89c5-4122ca0edd22\" (UID: \"1fc8dc61-d744-4c4b-89c5-4122ca0edd22\") "
Oct 30 01:20:54.015781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:54.015624    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-kube-api-access-7kjxh\" (UniqueName: \"kubernetes.io/projected/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-kube-api-access-7kjxh\") pod \"1fc8dc61-d744-4c4b-89c5-4122ca0edd22\" (UID: \"1fc8dc61-d744-4c4b-89c5-4122ca0edd22\") "
Oct 30 01:20:54.015997 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:54.015950    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-maas-api-openshift-ca" pod "1fc8dc61-d744-4c4b-89c5-4122ca0edd22" (UID: "1fc8dc61-d744-4c4b-89c5-4122ca0edd22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:54.016566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:54.016545    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:54.018310 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:54.018279    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-maas-api-xks-ca" pod "1fc8dc61-d744-4c4b-89c5-4122ca0edd22" (UID: "1fc8dc61-d744-4c4b-89c5-4122ca0edd22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:54.018435 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:54.018308    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-kube-api-access-7kjxh" pod "1fc8dc61-d744-4c4b-89c5-4122ca0edd22" (UID: "1fc8dc61-d744-4c4b-89c5-4122ca0edd22"). InnerVolumeSpecName "kube-api-access-7kjxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:54.117693 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:54.117661    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kjxh\" (UniqueName: \"kubernetes.io/projected/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-kube-api-access-7kjxh\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:54.117693 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:54.117690    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fc8dc61-d744-4c4b-89c5-4122ca0edd22-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:20:54.759979 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:54.759947    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7" event={"ID":"1fc8dc61-d744-4c4b-89c5-4122ca0edd22","Type":"ContainerDied","Data":"01634ad4ad41d7d5d6b258bfccfe179db7c305fd2e35c7ad53e05041e3727bb2"}
Oct 30 01:20:54.760413 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:54.760007    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01634ad4ad41d7d5d6b258bfccfe179db7c305fd2e35c7ad53e05041e3727bb2"
Oct 30 01:20:54.760413 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:54.759954    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7"
Oct 30 01:20:55.348598 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:55.348562    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"]
Oct 30 01:20:55.348864 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:55.348818    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2" podUID="caf7a449-3657-4b2e-aaa6-2cca317f241f" containerName="istio-proxy" containerID="cri-o://eff1c434d7c4aea6ad5c329f28a96010e88a2c97e1c894d64f4e818db5eee2de" gracePeriod=30
Oct 30 01:20:55.382142 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:55.382112    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:20:55.382308 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:20:55.382157    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2" podUID="caf7a449-3657-4b2e-aaa6-2cca317f241f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:21:00.677037 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.677012    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:21:00.727566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.727534    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf7a449-3657-4b2e-aaa6-2cca317f241f-kube-api-access-5v29s\" (UniqueName: \"kubernetes.io/projected/caf7a449-3657-4b2e-aaa6-2cca317f241f-kube-api-access-5v29s\") pod \"caf7a449-3657-4b2e-aaa6-2cca317f241f\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") "
Oct 30 01:21:00.727747 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.727603    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-workload-certs\") pod \"caf7a449-3657-4b2e-aaa6-2cca317f241f\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") "
Oct 30 01:21:00.727747 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.727701    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-envoy\") pod \"caf7a449-3657-4b2e-aaa6-2cca317f241f\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") "
Oct 30 01:21:00.727850 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.727767    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-workload-socket\") pod \"caf7a449-3657-4b2e-aaa6-2cca317f241f\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") "
Oct 30 01:21:00.727850 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.727814    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-credential-socket\") pod \"caf7a449-3657-4b2e-aaa6-2cca317f241f\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") "
Oct 30 01:21:00.727850 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.727846    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-data\") pod \"caf7a449-3657-4b2e-aaa6-2cca317f241f\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") "
Oct 30 01:21:00.728022 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.727870    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-workload-certs" pod "caf7a449-3657-4b2e-aaa6-2cca317f241f" (UID: "caf7a449-3657-4b2e-aaa6-2cca317f241f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:00.728022 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.727943    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-podinfo\") pod \"caf7a449-3657-4b2e-aaa6-2cca317f241f\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") "
Oct 30 01:21:00.728022 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.728013    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-token\" (UniqueName: \"kubernetes.io/projected/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-token\") pod \"caf7a449-3657-4b2e-aaa6-2cca317f241f\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") "
Oct 30 01:21:00.728208 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.728076    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-workload-socket" pod "caf7a449-3657-4b2e-aaa6-2cca317f241f" (UID: "caf7a449-3657-4b2e-aaa6-2cca317f241f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:00.728208 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.728109    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-credential-socket" pod "caf7a449-3657-4b2e-aaa6-2cca317f241f" (UID: "caf7a449-3657-4b2e-aaa6-2cca317f241f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:00.728208 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.728119    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-data" pod "caf7a449-3657-4b2e-aaa6-2cca317f241f" (UID: "caf7a449-3657-4b2e-aaa6-2cca317f241f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:00.728384 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.728319    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf7a449-3657-4b2e-aaa6-2cca317f241f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf7a449-3657-4b2e-aaa6-2cca317f241f-istiod-ca-cert\") pod \"caf7a449-3657-4b2e-aaa6-2cca317f241f\" (UID: \"caf7a449-3657-4b2e-aaa6-2cca317f241f\") "
Oct 30 01:21:00.728777 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.728750    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf7a449-3657-4b2e-aaa6-2cca317f241f-istiod-ca-cert" pod "caf7a449-3657-4b2e-aaa6-2cca317f241f" (UID: "caf7a449-3657-4b2e-aaa6-2cca317f241f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:00.729317 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.729292    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf7a449-3657-4b2e-aaa6-2cca317f241f-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:00.729413 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.729323    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:00.729413 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.729336    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:00.729413 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.729350    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:00.729413 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.729364    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:00.730151 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.730122    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf7a449-3657-4b2e-aaa6-2cca317f241f-kube-api-access-5v29s" pod "caf7a449-3657-4b2e-aaa6-2cca317f241f" (UID: "caf7a449-3657-4b2e-aaa6-2cca317f241f"). InnerVolumeSpecName "kube-api-access-5v29s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:00.730151 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.730122    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-envoy" pod "caf7a449-3657-4b2e-aaa6-2cca317f241f" (UID: "caf7a449-3657-4b2e-aaa6-2cca317f241f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:00.730312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.730165    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-token" pod "caf7a449-3657-4b2e-aaa6-2cca317f241f" (UID: "caf7a449-3657-4b2e-aaa6-2cca317f241f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:00.730312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.730199    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-podinfo" pod "caf7a449-3657-4b2e-aaa6-2cca317f241f" (UID: "caf7a449-3657-4b2e-aaa6-2cca317f241f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:21:00.809700 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.809665    6099 generic.go:356] "Generic (PLEG): container finished" podID="caf7a449-3657-4b2e-aaa6-2cca317f241f" containerID="eff1c434d7c4aea6ad5c329f28a96010e88a2c97e1c894d64f4e818db5eee2de" exitCode=0
Oct 30 01:21:00.809922 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.809731    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"
Oct 30 01:21:00.809922 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.809754    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2" event={"ID":"caf7a449-3657-4b2e-aaa6-2cca317f241f","Type":"ContainerDied","Data":"eff1c434d7c4aea6ad5c329f28a96010e88a2c97e1c894d64f4e818db5eee2de"}
Oct 30 01:21:00.809922 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.809820    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2" event={"ID":"caf7a449-3657-4b2e-aaa6-2cca317f241f","Type":"ContainerDied","Data":"cd07493f4492ef89e007834f1e3165139306bc5ff1f1e0e5b28be6b6ab32c95f"}
Oct 30 01:21:00.809922 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.809852    6099 scope.go:122] "RemoveContainer" containerID="eff1c434d7c4aea6ad5c329f28a96010e88a2c97e1c894d64f4e818db5eee2de"
Oct 30 01:21:00.822911 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.822892    6099 scope.go:122] "RemoveContainer" containerID="c35efa95ab8313096506a40443269c2490bad57fae4fa5d348ce8f09e728f8b9"
Oct 30 01:21:00.831065 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.831043    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v29s\" (UniqueName: \"kubernetes.io/projected/caf7a449-3657-4b2e-aaa6-2cca317f241f-kube-api-access-5v29s\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:00.831065 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.831067    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:00.831197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.831078    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:00.831197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.831087    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf7a449-3657-4b2e-aaa6-2cca317f241f-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:00.834300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.834287    6099 scope.go:122] "RemoveContainer" containerID="eff1c434d7c4aea6ad5c329f28a96010e88a2c97e1c894d64f4e818db5eee2de"
Oct 30 01:21:00.834560 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:21:00.834541    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eff1c434d7c4aea6ad5c329f28a96010e88a2c97e1c894d64f4e818db5eee2de\": container with ID starting with eff1c434d7c4aea6ad5c329f28a96010e88a2c97e1c894d64f4e818db5eee2de not found: ID does not exist" containerID="eff1c434d7c4aea6ad5c329f28a96010e88a2c97e1c894d64f4e818db5eee2de"
Oct 30 01:21:00.834603 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.834568    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff1c434d7c4aea6ad5c329f28a96010e88a2c97e1c894d64f4e818db5eee2de"} err="failed to get container status \"eff1c434d7c4aea6ad5c329f28a96010e88a2c97e1c894d64f4e818db5eee2de\": rpc error: code = NotFound desc = could not find container \"eff1c434d7c4aea6ad5c329f28a96010e88a2c97e1c894d64f4e818db5eee2de\": container with ID starting with eff1c434d7c4aea6ad5c329f28a96010e88a2c97e1c894d64f4e818db5eee2de not found: ID does not exist"
Oct 30 01:21:00.834603 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.834584    6099 scope.go:122] "RemoveContainer" containerID="c35efa95ab8313096506a40443269c2490bad57fae4fa5d348ce8f09e728f8b9"
Oct 30 01:21:00.834832 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:21:00.834816    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c35efa95ab8313096506a40443269c2490bad57fae4fa5d348ce8f09e728f8b9\": container with ID starting with c35efa95ab8313096506a40443269c2490bad57fae4fa5d348ce8f09e728f8b9 not found: ID does not exist" containerID="c35efa95ab8313096506a40443269c2490bad57fae4fa5d348ce8f09e728f8b9"
Oct 30 01:21:00.834882 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.834836    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c35efa95ab8313096506a40443269c2490bad57fae4fa5d348ce8f09e728f8b9"} err="failed to get container status \"c35efa95ab8313096506a40443269c2490bad57fae4fa5d348ce8f09e728f8b9\": rpc error: code = NotFound desc = could not find container \"c35efa95ab8313096506a40443269c2490bad57fae4fa5d348ce8f09e728f8b9\": container with ID starting with c35efa95ab8313096506a40443269c2490bad57fae4fa5d348ce8f09e728f8b9 not found: ID does not exist"
Oct 30 01:21:00.862454 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.862430    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"]
Oct 30 01:21:00.865837 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:00.865818    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c4f6f-openshift-default-6bbb9d4f57-q44f2"]
Oct 30 01:21:02.288727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:02.285430    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf7a449-3657-4b2e-aaa6-2cca317f241f" path="/var/lib/kubelet/pods/caf7a449-3657-4b2e-aaa6-2cca317f241f/volumes"
Oct 30 01:21:06.046769 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:06.046728    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4"]
Oct 30 01:21:06.094340 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:06.094301    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v"]
Oct 30 01:21:07.964497 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:07.964469    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7"]
Oct 30 01:21:07.993729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:07.993132    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bea3ef-zh9c7"]
Oct 30 01:21:08.264239 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:08.264150    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fc8dc61-d744-4c4b-89c5-4122ca0edd22" path="/var/lib/kubelet/pods/1fc8dc61-d744-4c4b-89c5-4122ca0edd22/volumes"
Oct 30 01:21:10.824118 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:10.824087    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"]
Oct 30 01:21:10.824513 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:10.824286    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2" podUID="7c1533e0-8ff9-4108-8688-714e05957447" containerName="istio-proxy" containerID="cri-o://743d37801d77c41b9983afcc6580305757b60c256a94a7d43968574a9bafe0ae" gracePeriod=30
Oct 30 01:21:11.047207 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:11.047167    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4" podUID="28560cb7-7b8c-46ba-bf1a-afef3f320722" containerName="payload-processing" containerID="cri-o://e445628a552d272537657c320d0d158d638c40ca3a4008074e03a038bd939e4d" gracePeriod=25
Oct 30 01:21:11.094834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:11.094739    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v" podUID="605336de-7c5e-42ed-bc77-2f6d56507518" containerName="payload-pre-processing" containerID="cri-o://3859af8c184f6528f99344ad9bbcb868d5def482907498a3c1cdae5531b478e0" gracePeriod=25
Oct 30 01:21:14.987274 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:14.987243    6099 generic.go:356] "Generic (PLEG): container finished" podID="51e427c9-5c89-4401-afa6-437676618dbf" containerID="509c1ad84607be8d861219d828745769d199bccd3408f168903ad43a347af7d8" exitCode=0
Oct 30 01:21:14.987665 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:14.987309    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp" event={"ID":"51e427c9-5c89-4401-afa6-437676618dbf","Type":"ContainerDied","Data":"509c1ad84607be8d861219d828745769d199bccd3408f168903ad43a347af7d8"}
Oct 30 01:21:14.987779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:14.987765    6099 scope.go:122] "RemoveContainer" containerID="509c1ad84607be8d861219d828745769d199bccd3408f168903ad43a347af7d8"
Oct 30 01:21:14.992455 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:14.992427    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:21:14.992558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:14.992517    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:21:14.992631 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:14.992567    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:21:15.380185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:15.380149    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"]
Oct 30 01:21:15.380411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:15.380385    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n" podUID="21a3cb0b-00bf-4b7b-b4cb-976310413ee1" containerName="istio-proxy" containerID="cri-o://b98ec03481824c514dc2f9a8cb0b26915c33aceba6e8a4395db92c787e2ef683" gracePeriod=30
Oct 30 01:21:15.998231 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:15.998176    6099 generic.go:356] "Generic (PLEG): container finished" podID="7c1533e0-8ff9-4108-8688-714e05957447" containerID="743d37801d77c41b9983afcc6580305757b60c256a94a7d43968574a9bafe0ae" exitCode=0
Oct 30 01:21:15.998949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:15.998310    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2" event={"ID":"7c1533e0-8ff9-4108-8688-714e05957447","Type":"ContainerDied","Data":"743d37801d77c41b9983afcc6580305757b60c256a94a7d43968574a9bafe0ae"}
Oct 30 01:21:16.000962 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.000933    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp" event={"ID":"51e427c9-5c89-4401-afa6-437676618dbf","Type":"ContainerStarted","Data":"4f6a2e2707560db95c8166f11273556c88045fc6624f92f41dc7043dcc5bea0f"}
Oct 30 01:21:16.107370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.107307    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:21:16.198402 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.198371    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-istio-data\") pod \"7c1533e0-8ff9-4108-8688-714e05957447\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") "
Oct 30 01:21:16.198597 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.198458    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c1533e0-8ff9-4108-8688-714e05957447-istio-token\" (UniqueName: \"kubernetes.io/projected/7c1533e0-8ff9-4108-8688-714e05957447-istio-token\") pod \"7c1533e0-8ff9-4108-8688-714e05957447\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") "
Oct 30 01:21:16.198597 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.198532    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-credential-socket\") pod \"7c1533e0-8ff9-4108-8688-714e05957447\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") "
Oct 30 01:21:16.198597 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.198555    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c1533e0-8ff9-4108-8688-714e05957447-kube-api-access-dpprh\" (UniqueName: \"kubernetes.io/projected/7c1533e0-8ff9-4108-8688-714e05957447-kube-api-access-dpprh\") pod \"7c1533e0-8ff9-4108-8688-714e05957447\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") "
Oct 30 01:21:16.198597 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.198587    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-workload-certs\") pod \"7c1533e0-8ff9-4108-8688-714e05957447\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") "
Oct 30 01:21:16.198901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.198646    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-workload-socket\") pod \"7c1533e0-8ff9-4108-8688-714e05957447\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") "
Oct 30 01:21:16.198901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.198663    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-istio-data" pod "7c1533e0-8ff9-4108-8688-714e05957447" (UID: "7c1533e0-8ff9-4108-8688-714e05957447"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:16.198901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.198688    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c1533e0-8ff9-4108-8688-714e05957447-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1533e0-8ff9-4108-8688-714e05957447-istiod-ca-cert\") pod \"7c1533e0-8ff9-4108-8688-714e05957447\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") "
Oct 30 01:21:16.198901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.198789    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c1533e0-8ff9-4108-8688-714e05957447-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1533e0-8ff9-4108-8688-714e05957447-istio-podinfo\") pod \"7c1533e0-8ff9-4108-8688-714e05957447\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") "
Oct 30 01:21:16.198901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.198877    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-istio-envoy\") pod \"7c1533e0-8ff9-4108-8688-714e05957447\" (UID: \"7c1533e0-8ff9-4108-8688-714e05957447\") "
Oct 30 01:21:16.199255 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.198902    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-credential-socket" pod "7c1533e0-8ff9-4108-8688-714e05957447" (UID: "7c1533e0-8ff9-4108-8688-714e05957447"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:16.199255 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.198954    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-workload-certs" pod "7c1533e0-8ff9-4108-8688-714e05957447" (UID: "7c1533e0-8ff9-4108-8688-714e05957447"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:16.199255 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.199243    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1533e0-8ff9-4108-8688-714e05957447-istiod-ca-cert" pod "7c1533e0-8ff9-4108-8688-714e05957447" (UID: "7c1533e0-8ff9-4108-8688-714e05957447"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:16.199479 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.199460    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-workload-socket" pod "7c1533e0-8ff9-4108-8688-714e05957447" (UID: "7c1533e0-8ff9-4108-8688-714e05957447"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:16.199720 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.199685    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:16.199720 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.199718    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:16.199900 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.199732    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:16.199900 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.199744    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1533e0-8ff9-4108-8688-714e05957447-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:16.199900 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.199756    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:16.201910 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.201876    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c1533e0-8ff9-4108-8688-714e05957447-istio-podinfo" pod "7c1533e0-8ff9-4108-8688-714e05957447" (UID: "7c1533e0-8ff9-4108-8688-714e05957447"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:21:16.202026 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.201914    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1533e0-8ff9-4108-8688-714e05957447-istio-token" pod "7c1533e0-8ff9-4108-8688-714e05957447" (UID: "7c1533e0-8ff9-4108-8688-714e05957447"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:16.202026 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.202016    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1533e0-8ff9-4108-8688-714e05957447-kube-api-access-dpprh" pod "7c1533e0-8ff9-4108-8688-714e05957447" (UID: "7c1533e0-8ff9-4108-8688-714e05957447"). InnerVolumeSpecName "kube-api-access-dpprh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:16.203497 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.203478    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-istio-envoy" pod "7c1533e0-8ff9-4108-8688-714e05957447" (UID: "7c1533e0-8ff9-4108-8688-714e05957447"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:16.301331 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.301300    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1533e0-8ff9-4108-8688-714e05957447-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:16.301331 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.301327    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpprh\" (UniqueName: \"kubernetes.io/projected/7c1533e0-8ff9-4108-8688-714e05957447-kube-api-access-dpprh\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:16.301544 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.301337    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1533e0-8ff9-4108-8688-714e05957447-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:16.301544 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.301346    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1533e0-8ff9-4108-8688-714e05957447-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:16.555269 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.555234    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:21:16.555463 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.555378    6099 patch_prober.go:29] interesting pod/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body=
Oct 30 01:21:16.555463 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:16.555419    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp" podUID="51e427c9-5c89-4401-afa6-437676618dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused"
Oct 30 01:21:16.630232 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:21:16.630193    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:21:17.011530 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:17.011449    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"
Oct 30 01:21:17.011530 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:17.011482    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2" event={"ID":"7c1533e0-8ff9-4108-8688-714e05957447","Type":"ContainerDied","Data":"afd7cad66432ecbdc5614084a409b36c82ec486c4c978ef474545f29f1368323"}
Oct 30 01:21:17.012064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:17.011544    6099 scope.go:122] "RemoveContainer" containerID="743d37801d77c41b9983afcc6580305757b60c256a94a7d43968574a9bafe0ae"
Oct 30 01:21:17.112568 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:17.112495    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"]
Oct 30 01:21:17.121299 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:17.121234    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-4ngr2"]
Oct 30 01:21:17.554876 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:17.554839    6099 patch_prober.go:29] interesting pod/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body=
Oct 30 01:21:17.555091 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:17.554894    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp" podUID="51e427c9-5c89-4401-afa6-437676618dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused"
Oct 30 01:21:18.298406 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:18.298355    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c1533e0-8ff9-4108-8688-714e05957447" path="/var/lib/kubelet/pods/7c1533e0-8ff9-4108-8688-714e05957447/volumes"
Oct 30 01:21:18.555094 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:18.555007    6099 patch_prober.go:29] interesting pod/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body=
Oct 30 01:21:18.555094 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:18.555068    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp" podUID="51e427c9-5c89-4401-afa6-437676618dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused"
Oct 30 01:21:19.508780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:19.508743    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v" podUID="605336de-7c5e-42ed-bc77-2f6d56507518" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:21:19.509160 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:19.508819    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4" podUID="28560cb7-7b8c-46ba-bf1a-afef3f320722" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:21:19.554760 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:19.554693    6099 patch_prober.go:29] interesting pod/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body=
Oct 30 01:21:19.554939 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:19.554783    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp" podUID="51e427c9-5c89-4401-afa6-437676618dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused"
Oct 30 01:21:20.555456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.555416    6099 patch_prober.go:29] interesting pod/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body=
Oct 30 01:21:20.555820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.555469    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp" podUID="51e427c9-5c89-4401-afa6-437676618dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused"
Oct 30 01:21:20.713190 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.713165    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 01:21:20.902221 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.902185    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-workload-certs\") pod \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") "
Oct 30 01:21:20.902472 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.902303    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-data\") pod \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") "
Oct 30 01:21:20.902472 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.902359    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-kube-api-access-lrmvx\" (UniqueName: \"kubernetes.io/projected/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-kube-api-access-lrmvx\") pod \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") "
Oct 30 01:21:20.902472 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.902443    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-workload-socket\") pod \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") "
Oct 30 01:21:20.902646 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.902484    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istiod-ca-cert\") pod \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") "
Oct 30 01:21:20.902646 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.902508    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-workload-certs" pod "21a3cb0b-00bf-4b7b-b4cb-976310413ee1" (UID: "21a3cb0b-00bf-4b7b-b4cb-976310413ee1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:20.902646 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.902523    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-token\" (UniqueName: \"kubernetes.io/projected/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-token\") pod \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") "
Oct 30 01:21:20.902646 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.902630    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-data" pod "21a3cb0b-00bf-4b7b-b4cb-976310413ee1" (UID: "21a3cb0b-00bf-4b7b-b4cb-976310413ee1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:20.902848 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.902640    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-podinfo\") pod \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") "
Oct 30 01:21:20.902848 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.902830    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-envoy\") pod \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") "
Oct 30 01:21:20.902959 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.902925    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-credential-socket\") pod \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\" (UID: \"21a3cb0b-00bf-4b7b-b4cb-976310413ee1\") "
Oct 30 01:21:20.903028 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.903006    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-workload-socket" pod "21a3cb0b-00bf-4b7b-b4cb-976310413ee1" (UID: "21a3cb0b-00bf-4b7b-b4cb-976310413ee1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:20.903210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.903187    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istiod-ca-cert" pod "21a3cb0b-00bf-4b7b-b4cb-976310413ee1" (UID: "21a3cb0b-00bf-4b7b-b4cb-976310413ee1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:20.903326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.903305    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-credential-socket" pod "21a3cb0b-00bf-4b7b-b4cb-976310413ee1" (UID: "21a3cb0b-00bf-4b7b-b4cb-976310413ee1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:20.903826 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.903804    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.903826 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.903825    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.903963 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.903839    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.903963 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.903851    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.903963 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.903863    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.904946 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.904926    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-podinfo" pod "21a3cb0b-00bf-4b7b-b4cb-976310413ee1" (UID: "21a3cb0b-00bf-4b7b-b4cb-976310413ee1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:21:20.905212 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.905185    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-token" pod "21a3cb0b-00bf-4b7b-b4cb-976310413ee1" (UID: "21a3cb0b-00bf-4b7b-b4cb-976310413ee1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:20.905312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.905219    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-kube-api-access-lrmvx" pod "21a3cb0b-00bf-4b7b-b4cb-976310413ee1" (UID: "21a3cb0b-00bf-4b7b-b4cb-976310413ee1"). InnerVolumeSpecName "kube-api-access-lrmvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:20.905312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:20.905256    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-envoy" pod "21a3cb0b-00bf-4b7b-b4cb-976310413ee1" (UID: "21a3cb0b-00bf-4b7b-b4cb-976310413ee1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:21.005652 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:21.005618    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:21.005652 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:21.005648    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:21.005652 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:21.005657    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:21.005907 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:21.005667    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrmvx\" (UniqueName: \"kubernetes.io/projected/21a3cb0b-00bf-4b7b-b4cb-976310413ee1-kube-api-access-lrmvx\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:21.045494 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:21.045458    6099 generic.go:356] "Generic (PLEG): container finished" podID="21a3cb0b-00bf-4b7b-b4cb-976310413ee1" containerID="b98ec03481824c514dc2f9a8cb0b26915c33aceba6e8a4395db92c787e2ef683" exitCode=0
Oct 30 01:21:21.045657 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:21.045507    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n" event={"ID":"21a3cb0b-00bf-4b7b-b4cb-976310413ee1","Type":"ContainerDied","Data":"b98ec03481824c514dc2f9a8cb0b26915c33aceba6e8a4395db92c787e2ef683"}
Oct 30 01:21:21.045657 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:21.045515    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"
Oct 30 01:21:21.045657 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:21.045559    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n" event={"ID":"21a3cb0b-00bf-4b7b-b4cb-976310413ee1","Type":"ContainerDied","Data":"38eeae654e910ed2aabdea434f5935a3363e4e6a5019f11ab51110a9e185de42"}
Oct 30 01:21:21.045657 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:21.045591    6099 scope.go:122] "RemoveContainer" containerID="b98ec03481824c514dc2f9a8cb0b26915c33aceba6e8a4395db92c787e2ef683"
Oct 30 01:21:21.058061 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:21.058041    6099 scope.go:122] "RemoveContainer" containerID="5c286554dde69bbbf0c6e25c1cf1be7a3d46897f8996de161f1ad54216f76dd4"
Oct 30 01:21:21.094307 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:21.094280    6099 scope.go:122] "RemoveContainer" containerID="b98ec03481824c514dc2f9a8cb0b26915c33aceba6e8a4395db92c787e2ef683"
Oct 30 01:21:21.094623 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:21:21.094599    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98ec03481824c514dc2f9a8cb0b26915c33aceba6e8a4395db92c787e2ef683\": container with ID starting with b98ec03481824c514dc2f9a8cb0b26915c33aceba6e8a4395db92c787e2ef683 not found: ID does not exist" containerID="b98ec03481824c514dc2f9a8cb0b26915c33aceba6e8a4395db92c787e2ef683"
Oct 30 01:21:21.094672 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:21.094633    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98ec03481824c514dc2f9a8cb0b26915c33aceba6e8a4395db92c787e2ef683"} err="failed to get container status \"b98ec03481824c514dc2f9a8cb0b26915c33aceba6e8a4395db92c787e2ef683\": rpc error: code = NotFound desc = could not find container \"b98ec03481824c514dc2f9a8cb0b26915c33aceba6e8a4395db92c787e2ef683\": container with ID starting with b98ec03481824c514dc2f9a8cb0b26915c33aceba6e8a4395db92c787e2ef683 not found: ID does not exist"
Oct 30 01:21:21.094672 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:21.094651    6099 scope.go:122] "RemoveContainer" containerID="5c286554dde69bbbf0c6e25c1cf1be7a3d46897f8996de161f1ad54216f76dd4"
Oct 30 01:21:21.094964 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:21:21.094947    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c286554dde69bbbf0c6e25c1cf1be7a3d46897f8996de161f1ad54216f76dd4\": container with ID starting with 5c286554dde69bbbf0c6e25c1cf1be7a3d46897f8996de161f1ad54216f76dd4 not found: ID does not exist" containerID="5c286554dde69bbbf0c6e25c1cf1be7a3d46897f8996de161f1ad54216f76dd4"
Oct 30 01:21:21.095039 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:21.094970    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c286554dde69bbbf0c6e25c1cf1be7a3d46897f8996de161f1ad54216f76dd4"} err="failed to get container status \"5c286554dde69bbbf0c6e25c1cf1be7a3d46897f8996de161f1ad54216f76dd4\": rpc error: code = NotFound desc = could not find container \"5c286554dde69bbbf0c6e25c1cf1be7a3d46897f8996de161f1ad54216f76dd4\": container with ID starting with 5c286554dde69bbbf0c6e25c1cf1be7a3d46897f8996de161f1ad54216f76dd4 not found: ID does not exist"
Oct 30 01:21:21.103197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:21.103175    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"]
Oct 30 01:21:21.116831 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:21.116806    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bea3ef-openshift-default-784b4db987-5w62n"]
Oct 30 01:21:21.554925 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:21.554895    6099 patch_prober.go:29] interesting pod/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body=
Oct 30 01:21:21.555115 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:21.554941    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp" podUID="51e427c9-5c89-4401-afa6-437676618dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused"
Oct 30 01:21:22.276014 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:22.275976    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21a3cb0b-00bf-4b7b-b4cb-976310413ee1" path="/var/lib/kubelet/pods/21a3cb0b-00bf-4b7b-b4cb-976310413ee1/volumes"
Oct 30 01:21:22.489224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:22.489183    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:21:22.616686 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:22.616650    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:21:23.061827 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:23.061799    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:21:29.508151 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:29.508114    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4" podUID="28560cb7-7b8c-46ba-bf1a-afef3f320722" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:21:29.508613 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:29.508254    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v" podUID="605336de-7c5e-42ed-bc77-2f6d56507518" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:21:31.397403 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:31.397365    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4"
Oct 30 01:21:31.495112 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:31.495089    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v"
Oct 30 01:21:31.512845 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:31.512816    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28560cb7-7b8c-46ba-bf1a-afef3f320722-kube-api-access-9tj22\" (UniqueName: \"kubernetes.io/projected/28560cb7-7b8c-46ba-bf1a-afef3f320722-kube-api-access-9tj22\") pod \"28560cb7-7b8c-46ba-bf1a-afef3f320722\" (UID: \"28560cb7-7b8c-46ba-bf1a-afef3f320722\") "
Oct 30 01:21:31.513032 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:31.513014    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28560cb7-7b8c-46ba-bf1a-afef3f320722-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28560cb7-7b8c-46ba-bf1a-afef3f320722-plugins-config-volume\") pod \"28560cb7-7b8c-46ba-bf1a-afef3f320722\" (UID: \"28560cb7-7b8c-46ba-bf1a-afef3f320722\") "
Oct 30 01:21:31.523724 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:31.515326    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28560cb7-7b8c-46ba-bf1a-afef3f320722-plugins-config-volume" pod "28560cb7-7b8c-46ba-bf1a-afef3f320722" (UID: "28560cb7-7b8c-46ba-bf1a-afef3f320722"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:31.523724 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:31.517365    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28560cb7-7b8c-46ba-bf1a-afef3f320722-kube-api-access-9tj22" pod "28560cb7-7b8c-46ba-bf1a-afef3f320722" (UID: "28560cb7-7b8c-46ba-bf1a-afef3f320722"). InnerVolumeSpecName "kube-api-access-9tj22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:31.616448 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:31.616371    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/605336de-7c5e-42ed-bc77-2f6d56507518-kube-api-access-wbztr\" (UniqueName: \"kubernetes.io/projected/605336de-7c5e-42ed-bc77-2f6d56507518-kube-api-access-wbztr\") pod \"605336de-7c5e-42ed-bc77-2f6d56507518\" (UID: \"605336de-7c5e-42ed-bc77-2f6d56507518\") "
Oct 30 01:21:31.616633 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:31.616473    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/605336de-7c5e-42ed-bc77-2f6d56507518-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605336de-7c5e-42ed-bc77-2f6d56507518-plugins-config-volume\") pod \"605336de-7c5e-42ed-bc77-2f6d56507518\" (UID: \"605336de-7c5e-42ed-bc77-2f6d56507518\") "
Oct 30 01:21:31.616867 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:31.616841    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605336de-7c5e-42ed-bc77-2f6d56507518-plugins-config-volume" pod "605336de-7c5e-42ed-bc77-2f6d56507518" (UID: "605336de-7c5e-42ed-bc77-2f6d56507518"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:31.617500 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:31.617199    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tj22\" (UniqueName: \"kubernetes.io/projected/28560cb7-7b8c-46ba-bf1a-afef3f320722-kube-api-access-9tj22\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:31.617500 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:31.617213    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28560cb7-7b8c-46ba-bf1a-afef3f320722-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:31.617500 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:31.617222    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605336de-7c5e-42ed-bc77-2f6d56507518-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:31.618531 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:31.618507    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605336de-7c5e-42ed-bc77-2f6d56507518-kube-api-access-wbztr" pod "605336de-7c5e-42ed-bc77-2f6d56507518" (UID: "605336de-7c5e-42ed-bc77-2f6d56507518"). InnerVolumeSpecName "kube-api-access-wbztr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:31.724429 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:31.724380    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbztr\" (UniqueName: \"kubernetes.io/projected/605336de-7c5e-42ed-bc77-2f6d56507518-kube-api-access-wbztr\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:32.154909 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.154876    6099 generic.go:356] "Generic (PLEG): container finished" podID="28560cb7-7b8c-46ba-bf1a-afef3f320722" containerID="e445628a552d272537657c320d0d158d638c40ca3a4008074e03a038bd939e4d" exitCode=0
Oct 30 01:21:32.155083 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.154922    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4"
Oct 30 01:21:32.155083 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.154957    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4" event={"ID":"28560cb7-7b8c-46ba-bf1a-afef3f320722","Type":"ContainerDied","Data":"e445628a552d272537657c320d0d158d638c40ca3a4008074e03a038bd939e4d"}
Oct 30 01:21:32.155083 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.155048    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4" event={"ID":"28560cb7-7b8c-46ba-bf1a-afef3f320722","Type":"ContainerDied","Data":"dfb6a3669ee08710da99c53de97803d1bf723e8b3938820ef9db22feb67bebce"}
Oct 30 01:21:32.155083 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.155078    6099 scope.go:122] "RemoveContainer" containerID="e445628a552d272537657c320d0d158d638c40ca3a4008074e03a038bd939e4d"
Oct 30 01:21:32.156422 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.156401    6099 generic.go:356] "Generic (PLEG): container finished" podID="605336de-7c5e-42ed-bc77-2f6d56507518" containerID="3859af8c184f6528f99344ad9bbcb868d5def482907498a3c1cdae5531b478e0" exitCode=0
Oct 30 01:21:32.156501 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.156450    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v"
Oct 30 01:21:32.156501 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.156459    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v" event={"ID":"605336de-7c5e-42ed-bc77-2f6d56507518","Type":"ContainerDied","Data":"3859af8c184f6528f99344ad9bbcb868d5def482907498a3c1cdae5531b478e0"}
Oct 30 01:21:32.156501 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.156498    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v" event={"ID":"605336de-7c5e-42ed-bc77-2f6d56507518","Type":"ContainerDied","Data":"f9a77b61742094c60eec978b3a912b2114e09f2ccb64de1b0d9c86d2796fab04"}
Oct 30 01:21:32.167198 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.167185    6099 scope.go:122] "RemoveContainer" containerID="e445628a552d272537657c320d0d158d638c40ca3a4008074e03a038bd939e4d"
Oct 30 01:21:32.167443 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:21:32.167422    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e445628a552d272537657c320d0d158d638c40ca3a4008074e03a038bd939e4d\": container with ID starting with e445628a552d272537657c320d0d158d638c40ca3a4008074e03a038bd939e4d not found: ID does not exist" containerID="e445628a552d272537657c320d0d158d638c40ca3a4008074e03a038bd939e4d"
Oct 30 01:21:32.167487 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.167451    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e445628a552d272537657c320d0d158d638c40ca3a4008074e03a038bd939e4d"} err="failed to get container status \"e445628a552d272537657c320d0d158d638c40ca3a4008074e03a038bd939e4d\": rpc error: code = NotFound desc = could not find container \"e445628a552d272537657c320d0d158d638c40ca3a4008074e03a038bd939e4d\": container with ID starting with e445628a552d272537657c320d0d158d638c40ca3a4008074e03a038bd939e4d not found: ID does not exist"
Oct 30 01:21:32.167487 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.167467    6099 scope.go:122] "RemoveContainer" containerID="3859af8c184f6528f99344ad9bbcb868d5def482907498a3c1cdae5531b478e0"
Oct 30 01:21:32.177481 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.177462    6099 scope.go:122] "RemoveContainer" containerID="3859af8c184f6528f99344ad9bbcb868d5def482907498a3c1cdae5531b478e0"
Oct 30 01:21:32.177761 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:21:32.177742    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3859af8c184f6528f99344ad9bbcb868d5def482907498a3c1cdae5531b478e0\": container with ID starting with 3859af8c184f6528f99344ad9bbcb868d5def482907498a3c1cdae5531b478e0 not found: ID does not exist" containerID="3859af8c184f6528f99344ad9bbcb868d5def482907498a3c1cdae5531b478e0"
Oct 30 01:21:32.177837 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.177768    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3859af8c184f6528f99344ad9bbcb868d5def482907498a3c1cdae5531b478e0"} err="failed to get container status \"3859af8c184f6528f99344ad9bbcb868d5def482907498a3c1cdae5531b478e0\": rpc error: code = NotFound desc = could not find container \"3859af8c184f6528f99344ad9bbcb868d5def482907498a3c1cdae5531b478e0\": container with ID starting with 3859af8c184f6528f99344ad9bbcb868d5def482907498a3c1cdae5531b478e0 not found: ID does not exist"
Oct 30 01:21:32.206827 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.206800    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4"]
Oct 30 01:21:32.218151 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.218130    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-33e759-86f4b57dc4-fksx4"]
Oct 30 01:21:32.243740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.243700    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v"]
Oct 30 01:21:32.249546 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.249521    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-33e759-56db6984f9-tx88v"]
Oct 30 01:21:32.264082 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.264061    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28560cb7-7b8c-46ba-bf1a-afef3f320722" path="/var/lib/kubelet/pods/28560cb7-7b8c-46ba-bf1a-afef3f320722/volumes"
Oct 30 01:21:32.264518 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:32.264502    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="605336de-7c5e-42ed-bc77-2f6d56507518" path="/var/lib/kubelet/pods/605336de-7c5e-42ed-bc77-2f6d56507518/volumes"
Oct 30 01:21:44.182230 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.182193    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l"]
Oct 30 01:21:44.182843 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.182384    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l" podUID="b2e137cc-9cf8-4212-a7cc-0759867fd7ea" containerName="payload-processing" containerID="cri-o://bb217a75db40af03cb0e0a4df0bf97b5e7d8ebdbd495c7f1253b48e09b21911e" gracePeriod=30
Oct 30 01:21:44.214417 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.214382    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m"]
Oct 30 01:21:44.214673 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.214627    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m" podUID="1bf54661-aa8e-4d60-9981-16006d11e41a" containerName="payload-pre-processing" containerID="cri-o://7a087ac5a4d45b1e9c57b67368d89b4a2d5ca0de6494392955457d492d6d448e" gracePeriod=30
Oct 30 01:21:44.532683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.532652    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l"
Oct 30 01:21:44.653728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.650268    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e137cc-9cf8-4212-a7cc-0759867fd7ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e137cc-9cf8-4212-a7cc-0759867fd7ea-plugins-config-volume\") pod \"b2e137cc-9cf8-4212-a7cc-0759867fd7ea\" (UID: \"b2e137cc-9cf8-4212-a7cc-0759867fd7ea\") "
Oct 30 01:21:44.653728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.650356    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e137cc-9cf8-4212-a7cc-0759867fd7ea-kube-api-access-z5wnr\" (UniqueName: \"kubernetes.io/projected/b2e137cc-9cf8-4212-a7cc-0759867fd7ea-kube-api-access-z5wnr\") pod \"b2e137cc-9cf8-4212-a7cc-0759867fd7ea\" (UID: \"b2e137cc-9cf8-4212-a7cc-0759867fd7ea\") "
Oct 30 01:21:44.689986 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.689943    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e137cc-9cf8-4212-a7cc-0759867fd7ea-plugins-config-volume" pod "b2e137cc-9cf8-4212-a7cc-0759867fd7ea" (UID: "b2e137cc-9cf8-4212-a7cc-0759867fd7ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:44.693078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.692937    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m"
Oct 30 01:21:44.716586 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.716478    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e137cc-9cf8-4212-a7cc-0759867fd7ea-kube-api-access-z5wnr" pod "b2e137cc-9cf8-4212-a7cc-0759867fd7ea" (UID: "b2e137cc-9cf8-4212-a7cc-0759867fd7ea"). InnerVolumeSpecName "kube-api-access-z5wnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:44.764983 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.764950    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf54661-aa8e-4d60-9981-16006d11e41a-kube-api-access-b5z2x\" (UniqueName: \"kubernetes.io/projected/1bf54661-aa8e-4d60-9981-16006d11e41a-kube-api-access-b5z2x\") pod \"1bf54661-aa8e-4d60-9981-16006d11e41a\" (UID: \"1bf54661-aa8e-4d60-9981-16006d11e41a\") "
Oct 30 01:21:44.765242 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.765220    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bf54661-aa8e-4d60-9981-16006d11e41a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf54661-aa8e-4d60-9981-16006d11e41a-plugins-config-volume\") pod \"1bf54661-aa8e-4d60-9981-16006d11e41a\" (UID: \"1bf54661-aa8e-4d60-9981-16006d11e41a\") "
Oct 30 01:21:44.765604 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.765578    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf54661-aa8e-4d60-9981-16006d11e41a-plugins-config-volume" pod "1bf54661-aa8e-4d60-9981-16006d11e41a" (UID: "1bf54661-aa8e-4d60-9981-16006d11e41a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:44.765940 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.765922    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf54661-aa8e-4d60-9981-16006d11e41a-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:44.766019 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.765946    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e137cc-9cf8-4212-a7cc-0759867fd7ea-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:44.766019 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.765959    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5wnr\" (UniqueName: \"kubernetes.io/projected/b2e137cc-9cf8-4212-a7cc-0759867fd7ea-kube-api-access-z5wnr\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:44.767105 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.767084    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf54661-aa8e-4d60-9981-16006d11e41a-kube-api-access-b5z2x" pod "1bf54661-aa8e-4d60-9981-16006d11e41a" (UID: "1bf54661-aa8e-4d60-9981-16006d11e41a"). InnerVolumeSpecName "kube-api-access-b5z2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:44.867742 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.867695    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5z2x\" (UniqueName: \"kubernetes.io/projected/1bf54661-aa8e-4d60-9981-16006d11e41a-kube-api-access-b5z2x\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:44.994476 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.994391    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59"]
Oct 30 01:21:44.999458 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:44.999433    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-481249-dnj59"]
Oct 30 01:21:45.295935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:45.295901    6099 generic.go:356] "Generic (PLEG): container finished" podID="1bf54661-aa8e-4d60-9981-16006d11e41a" containerID="7a087ac5a4d45b1e9c57b67368d89b4a2d5ca0de6494392955457d492d6d448e" exitCode=0
Oct 30 01:21:45.296417 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:45.295969    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m"
Oct 30 01:21:45.296417 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:45.295987    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m" event={"ID":"1bf54661-aa8e-4d60-9981-16006d11e41a","Type":"ContainerDied","Data":"7a087ac5a4d45b1e9c57b67368d89b4a2d5ca0de6494392955457d492d6d448e"}
Oct 30 01:21:45.296417 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:45.296068    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m" event={"ID":"1bf54661-aa8e-4d60-9981-16006d11e41a","Type":"ContainerDied","Data":"74e0d62664712f7e3605bd23265a778ab1ee47c1b8ad6628c8d02f2357e2c5cd"}
Oct 30 01:21:45.296417 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:45.296099    6099 scope.go:122] "RemoveContainer" containerID="7a087ac5a4d45b1e9c57b67368d89b4a2d5ca0de6494392955457d492d6d448e"
Oct 30 01:21:45.298157 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:45.298130    6099 generic.go:356] "Generic (PLEG): container finished" podID="b2e137cc-9cf8-4212-a7cc-0759867fd7ea" containerID="bb217a75db40af03cb0e0a4df0bf97b5e7d8ebdbd495c7f1253b48e09b21911e" exitCode=0
Oct 30 01:21:45.298299 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:45.298166    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l"
Oct 30 01:21:45.298299 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:45.298210    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l" event={"ID":"b2e137cc-9cf8-4212-a7cc-0759867fd7ea","Type":"ContainerDied","Data":"bb217a75db40af03cb0e0a4df0bf97b5e7d8ebdbd495c7f1253b48e09b21911e"}
Oct 30 01:21:45.298299 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:45.298248    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l" event={"ID":"b2e137cc-9cf8-4212-a7cc-0759867fd7ea","Type":"ContainerDied","Data":"a404845e32c57cec94928f27d1060faff458f0f34b1320336eb5e8913537c68e"}
Oct 30 01:21:45.310053 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:45.310032    6099 scope.go:122] "RemoveContainer" containerID="7a087ac5a4d45b1e9c57b67368d89b4a2d5ca0de6494392955457d492d6d448e"
Oct 30 01:21:45.310415 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:21:45.310388    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a087ac5a4d45b1e9c57b67368d89b4a2d5ca0de6494392955457d492d6d448e\": container with ID starting with 7a087ac5a4d45b1e9c57b67368d89b4a2d5ca0de6494392955457d492d6d448e not found: ID does not exist" containerID="7a087ac5a4d45b1e9c57b67368d89b4a2d5ca0de6494392955457d492d6d448e"
Oct 30 01:21:45.310542 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:45.310459    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a087ac5a4d45b1e9c57b67368d89b4a2d5ca0de6494392955457d492d6d448e"} err="failed to get container status \"7a087ac5a4d45b1e9c57b67368d89b4a2d5ca0de6494392955457d492d6d448e\": rpc error: code = NotFound desc = could not find container \"7a087ac5a4d45b1e9c57b67368d89b4a2d5ca0de6494392955457d492d6d448e\": container with ID starting with 7a087ac5a4d45b1e9c57b67368d89b4a2d5ca0de6494392955457d492d6d448e not found: ID does not exist"
Oct 30 01:21:45.310542 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:45.310484    6099 scope.go:122] "RemoveContainer" containerID="bb217a75db40af03cb0e0a4df0bf97b5e7d8ebdbd495c7f1253b48e09b21911e"
Oct 30 01:21:45.320807 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:45.320793    6099 scope.go:122] "RemoveContainer" containerID="bb217a75db40af03cb0e0a4df0bf97b5e7d8ebdbd495c7f1253b48e09b21911e"
Oct 30 01:21:45.321109 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:21:45.321084    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb217a75db40af03cb0e0a4df0bf97b5e7d8ebdbd495c7f1253b48e09b21911e\": container with ID starting with bb217a75db40af03cb0e0a4df0bf97b5e7d8ebdbd495c7f1253b48e09b21911e not found: ID does not exist" containerID="bb217a75db40af03cb0e0a4df0bf97b5e7d8ebdbd495c7f1253b48e09b21911e"
Oct 30 01:21:45.321181 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:45.321121    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb217a75db40af03cb0e0a4df0bf97b5e7d8ebdbd495c7f1253b48e09b21911e"} err="failed to get container status \"bb217a75db40af03cb0e0a4df0bf97b5e7d8ebdbd495c7f1253b48e09b21911e\": rpc error: code = NotFound desc = could not find container \"bb217a75db40af03cb0e0a4df0bf97b5e7d8ebdbd495c7f1253b48e09b21911e\": container with ID starting with bb217a75db40af03cb0e0a4df0bf97b5e7d8ebdbd495c7f1253b48e09b21911e not found: ID does not exist"
Oct 30 01:21:45.361739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:45.361717    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m"]
Oct 30 01:21:45.371037 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:45.371008    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-481249-6686b74599-lw59m"]
Oct 30 01:21:45.407792 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:45.407757    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l"]
Oct 30 01:21:45.409860 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:45.409833    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-481249-6d59bcdc6b-f727l"]
Oct 30 01:21:46.266865 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:46.266834    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bf54661-aa8e-4d60-9981-16006d11e41a" path="/var/lib/kubelet/pods/1bf54661-aa8e-4d60-9981-16006d11e41a/volumes"
Oct 30 01:21:46.267393 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:46.267376    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41fd810f-4559-492b-8038-7cd547121618" path="/var/lib/kubelet/pods/41fd810f-4559-492b-8038-7cd547121618/volumes"
Oct 30 01:21:46.267880 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:46.267866    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e137cc-9cf8-4212-a7cc-0759867fd7ea" path="/var/lib/kubelet/pods/b2e137cc-9cf8-4212-a7cc-0759867fd7ea/volumes"
Oct 30 01:21:46.450396 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:46.450359    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"]
Oct 30 01:21:46.450993 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:46.450631    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6" podUID="75cc8389-4904-44b2-8c92-cf3debf351f4" containerName="maas-api" containerID="cri-o://6359d9b2d450528e76ccf389945b98abdca2178714bfa1d908e81beecb8f2027" gracePeriod=30
Oct 30 01:21:46.618413 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:46.616843    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m"]
Oct 30 01:21:46.618413 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:46.617138    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m" podUID="57a0acc3-77a4-4ad2-978e-08280840d903" containerName="payload-processing" containerID="cri-o://897ae639aa8f5e6562e39a7841af7bea83634e8c9ba584275e432d1c7c36f6f3" gracePeriod=30
Oct 30 01:21:46.660178 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:46.647787    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf"]
Oct 30 01:21:46.660178 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:46.648021    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf" podUID="c3dec5ee-7aba-4409-b10d-bed8d78c1df3" containerName="payload-pre-processing" containerID="cri-o://ee706cccfcba4bfb4c05cad3045032506f88dcdf7b69907b90847d5ade7a340d" gracePeriod=30
Oct 30 01:21:46.660178 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:21:46.658319    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:21:46.869540 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:46.868659    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"
Oct 30 01:21:47.098332 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.098295    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75cc8389-4904-44b2-8c92-cf3debf351f4-kube-api-access-wct6f\" (UniqueName: \"kubernetes.io/projected/75cc8389-4904-44b2-8c92-cf3debf351f4-kube-api-access-wct6f\") pod \"75cc8389-4904-44b2-8c92-cf3debf351f4\" (UID: \"75cc8389-4904-44b2-8c92-cf3debf351f4\") "
Oct 30 01:21:47.098532 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.098383    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75cc8389-4904-44b2-8c92-cf3debf351f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cc8389-4904-44b2-8c92-cf3debf351f4-openshift-service-ca\") pod \"75cc8389-4904-44b2-8c92-cf3debf351f4\" (UID: \"75cc8389-4904-44b2-8c92-cf3debf351f4\") "
Oct 30 01:21:47.098532 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.098418    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75cc8389-4904-44b2-8c92-cf3debf351f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75cc8389-4904-44b2-8c92-cf3debf351f4-maas-api-tls\") pod \"75cc8389-4904-44b2-8c92-cf3debf351f4\" (UID: \"75cc8389-4904-44b2-8c92-cf3debf351f4\") "
Oct 30 01:21:47.098785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.098749    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75cc8389-4904-44b2-8c92-cf3debf351f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cc8389-4904-44b2-8c92-cf3debf351f4-metrics-certs\") pod \"75cc8389-4904-44b2-8c92-cf3debf351f4\" (UID: \"75cc8389-4904-44b2-8c92-cf3debf351f4\") "
Oct 30 01:21:47.100810 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.100780    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75cc8389-4904-44b2-8c92-cf3debf351f4-openshift-service-ca" pod "75cc8389-4904-44b2-8c92-cf3debf351f4" (UID: "75cc8389-4904-44b2-8c92-cf3debf351f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:47.128933 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.120683    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75cc8389-4904-44b2-8c92-cf3debf351f4-metrics-certs" pod "75cc8389-4904-44b2-8c92-cf3debf351f4" (UID: "75cc8389-4904-44b2-8c92-cf3debf351f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:47.128933 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.124146    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75cc8389-4904-44b2-8c92-cf3debf351f4-maas-api-tls" pod "75cc8389-4904-44b2-8c92-cf3debf351f4" (UID: "75cc8389-4904-44b2-8c92-cf3debf351f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:47.128933 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.126324    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75cc8389-4904-44b2-8c92-cf3debf351f4-kube-api-access-wct6f" pod "75cc8389-4904-44b2-8c92-cf3debf351f4" (UID: "75cc8389-4904-44b2-8c92-cf3debf351f4"). InnerVolumeSpecName "kube-api-access-wct6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:47.129969 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.129941    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m"
Oct 30 01:21:47.169755 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.169684    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf"
Oct 30 01:21:47.200927 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.200892    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a0acc3-77a4-4ad2-978e-08280840d903-kube-api-access-x9k7f\" (UniqueName: \"kubernetes.io/projected/57a0acc3-77a4-4ad2-978e-08280840d903-kube-api-access-x9k7f\") pod \"57a0acc3-77a4-4ad2-978e-08280840d903\" (UID: \"57a0acc3-77a4-4ad2-978e-08280840d903\") "
Oct 30 01:21:47.201301 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.201278    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57a0acc3-77a4-4ad2-978e-08280840d903-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a0acc3-77a4-4ad2-978e-08280840d903-plugins-config-volume\") pod \"57a0acc3-77a4-4ad2-978e-08280840d903\" (UID: \"57a0acc3-77a4-4ad2-978e-08280840d903\") "
Oct 30 01:21:47.202479 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.201663    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a0acc3-77a4-4ad2-978e-08280840d903-plugins-config-volume" pod "57a0acc3-77a4-4ad2-978e-08280840d903" (UID: "57a0acc3-77a4-4ad2-978e-08280840d903"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:47.202479 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.202384    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wct6f\" (UniqueName: \"kubernetes.io/projected/75cc8389-4904-44b2-8c92-cf3debf351f4-kube-api-access-wct6f\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:47.202479 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.202410    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cc8389-4904-44b2-8c92-cf3debf351f4-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:47.202479 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.202424    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75cc8389-4904-44b2-8c92-cf3debf351f4-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:47.202479 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.202437    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cc8389-4904-44b2-8c92-cf3debf351f4-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:47.202479 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.202449    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a0acc3-77a4-4ad2-978e-08280840d903-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:47.204309 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.204284    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a0acc3-77a4-4ad2-978e-08280840d903-kube-api-access-x9k7f" pod "57a0acc3-77a4-4ad2-978e-08280840d903" (UID: "57a0acc3-77a4-4ad2-978e-08280840d903"). InnerVolumeSpecName "kube-api-access-x9k7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:47.303463 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.303430    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3dec5ee-7aba-4409-b10d-bed8d78c1df3-kube-api-access-zxk9q\" (UniqueName: \"kubernetes.io/projected/c3dec5ee-7aba-4409-b10d-bed8d78c1df3-kube-api-access-zxk9q\") pod \"c3dec5ee-7aba-4409-b10d-bed8d78c1df3\" (UID: \"c3dec5ee-7aba-4409-b10d-bed8d78c1df3\") "
Oct 30 01:21:47.303652 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.303634    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3dec5ee-7aba-4409-b10d-bed8d78c1df3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3dec5ee-7aba-4409-b10d-bed8d78c1df3-plugins-config-volume\") pod \"c3dec5ee-7aba-4409-b10d-bed8d78c1df3\" (UID: \"c3dec5ee-7aba-4409-b10d-bed8d78c1df3\") "
Oct 30 01:21:47.304036 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.304011    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3dec5ee-7aba-4409-b10d-bed8d78c1df3-plugins-config-volume" pod "c3dec5ee-7aba-4409-b10d-bed8d78c1df3" (UID: "c3dec5ee-7aba-4409-b10d-bed8d78c1df3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:47.304677 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.304653    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3dec5ee-7aba-4409-b10d-bed8d78c1df3-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:47.304784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.304678    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9k7f\" (UniqueName: \"kubernetes.io/projected/57a0acc3-77a4-4ad2-978e-08280840d903-kube-api-access-x9k7f\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:47.305771 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.305749    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3dec5ee-7aba-4409-b10d-bed8d78c1df3-kube-api-access-zxk9q" pod "c3dec5ee-7aba-4409-b10d-bed8d78c1df3" (UID: "c3dec5ee-7aba-4409-b10d-bed8d78c1df3"). InnerVolumeSpecName "kube-api-access-zxk9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:47.318124 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.318100    6099 generic.go:356] "Generic (PLEG): container finished" podID="75cc8389-4904-44b2-8c92-cf3debf351f4" containerID="6359d9b2d450528e76ccf389945b98abdca2178714bfa1d908e81beecb8f2027" exitCode=0
Oct 30 01:21:47.318234 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.318160    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"
Oct 30 01:21:47.318234 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.318179    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6" event={"ID":"75cc8389-4904-44b2-8c92-cf3debf351f4","Type":"ContainerDied","Data":"6359d9b2d450528e76ccf389945b98abdca2178714bfa1d908e81beecb8f2027"}
Oct 30 01:21:47.318335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.318243    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6" event={"ID":"75cc8389-4904-44b2-8c92-cf3debf351f4","Type":"ContainerDied","Data":"3ea62a5114b0c3da97e311f0f562d562682a676dccc7338ef6a936ef67ef05f7"}
Oct 30 01:21:47.318335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.318273    6099 scope.go:122] "RemoveContainer" containerID="6359d9b2d450528e76ccf389945b98abdca2178714bfa1d908e81beecb8f2027"
Oct 30 01:21:47.319772 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.319734    6099 generic.go:356] "Generic (PLEG): container finished" podID="57a0acc3-77a4-4ad2-978e-08280840d903" containerID="897ae639aa8f5e6562e39a7841af7bea83634e8c9ba584275e432d1c7c36f6f3" exitCode=0
Oct 30 01:21:47.319890 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.319768    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m" event={"ID":"57a0acc3-77a4-4ad2-978e-08280840d903","Type":"ContainerDied","Data":"897ae639aa8f5e6562e39a7841af7bea83634e8c9ba584275e432d1c7c36f6f3"}
Oct 30 01:21:47.319890 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.319796    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m"
Oct 30 01:21:47.319890 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.319832    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m" event={"ID":"57a0acc3-77a4-4ad2-978e-08280840d903","Type":"ContainerDied","Data":"e2a845639c73da83207d59ebddc821a72f3676c02318175268224d96c424b7c8"}
Oct 30 01:21:47.321802 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.321616    6099 generic.go:356] "Generic (PLEG): container finished" podID="c3dec5ee-7aba-4409-b10d-bed8d78c1df3" containerID="ee706cccfcba4bfb4c05cad3045032506f88dcdf7b69907b90847d5ade7a340d" exitCode=0
Oct 30 01:21:47.321802 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.321691    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf" event={"ID":"c3dec5ee-7aba-4409-b10d-bed8d78c1df3","Type":"ContainerDied","Data":"ee706cccfcba4bfb4c05cad3045032506f88dcdf7b69907b90847d5ade7a340d"}
Oct 30 01:21:47.321802 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.321750    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf" event={"ID":"c3dec5ee-7aba-4409-b10d-bed8d78c1df3","Type":"ContainerDied","Data":"23ce837a7f5cfb4c14814b2a46a2d66985eb0a1ec6c89335d8a0d98e6828f8a7"}
Oct 30 01:21:47.322174 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.321664    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf"
Oct 30 01:21:47.332421 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.332404    6099 scope.go:122] "RemoveContainer" containerID="6359d9b2d450528e76ccf389945b98abdca2178714bfa1d908e81beecb8f2027"
Oct 30 01:21:47.332653 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:21:47.332636    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6359d9b2d450528e76ccf389945b98abdca2178714bfa1d908e81beecb8f2027\": container with ID starting with 6359d9b2d450528e76ccf389945b98abdca2178714bfa1d908e81beecb8f2027 not found: ID does not exist" containerID="6359d9b2d450528e76ccf389945b98abdca2178714bfa1d908e81beecb8f2027"
Oct 30 01:21:47.332698 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.332659    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6359d9b2d450528e76ccf389945b98abdca2178714bfa1d908e81beecb8f2027"} err="failed to get container status \"6359d9b2d450528e76ccf389945b98abdca2178714bfa1d908e81beecb8f2027\": rpc error: code = NotFound desc = could not find container \"6359d9b2d450528e76ccf389945b98abdca2178714bfa1d908e81beecb8f2027\": container with ID starting with 6359d9b2d450528e76ccf389945b98abdca2178714bfa1d908e81beecb8f2027 not found: ID does not exist"
Oct 30 01:21:47.332698 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.332675    6099 scope.go:122] "RemoveContainer" containerID="897ae639aa8f5e6562e39a7841af7bea83634e8c9ba584275e432d1c7c36f6f3"
Oct 30 01:21:47.345641 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.345623    6099 scope.go:122] "RemoveContainer" containerID="897ae639aa8f5e6562e39a7841af7bea83634e8c9ba584275e432d1c7c36f6f3"
Oct 30 01:21:47.345923 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:21:47.345901    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897ae639aa8f5e6562e39a7841af7bea83634e8c9ba584275e432d1c7c36f6f3\": container with ID starting with 897ae639aa8f5e6562e39a7841af7bea83634e8c9ba584275e432d1c7c36f6f3 not found: ID does not exist" containerID="897ae639aa8f5e6562e39a7841af7bea83634e8c9ba584275e432d1c7c36f6f3"
Oct 30 01:21:47.346027 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.345933    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897ae639aa8f5e6562e39a7841af7bea83634e8c9ba584275e432d1c7c36f6f3"} err="failed to get container status \"897ae639aa8f5e6562e39a7841af7bea83634e8c9ba584275e432d1c7c36f6f3\": rpc error: code = NotFound desc = could not find container \"897ae639aa8f5e6562e39a7841af7bea83634e8c9ba584275e432d1c7c36f6f3\": container with ID starting with 897ae639aa8f5e6562e39a7841af7bea83634e8c9ba584275e432d1c7c36f6f3 not found: ID does not exist"
Oct 30 01:21:47.346027 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.345954    6099 scope.go:122] "RemoveContainer" containerID="ee706cccfcba4bfb4c05cad3045032506f88dcdf7b69907b90847d5ade7a340d"
Oct 30 01:21:47.356318 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.356300    6099 scope.go:122] "RemoveContainer" containerID="ee706cccfcba4bfb4c05cad3045032506f88dcdf7b69907b90847d5ade7a340d"
Oct 30 01:21:47.356618 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:21:47.356590    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee706cccfcba4bfb4c05cad3045032506f88dcdf7b69907b90847d5ade7a340d\": container with ID starting with ee706cccfcba4bfb4c05cad3045032506f88dcdf7b69907b90847d5ade7a340d not found: ID does not exist" containerID="ee706cccfcba4bfb4c05cad3045032506f88dcdf7b69907b90847d5ade7a340d"
Oct 30 01:21:47.356754 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.356626    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee706cccfcba4bfb4c05cad3045032506f88dcdf7b69907b90847d5ade7a340d"} err="failed to get container status \"ee706cccfcba4bfb4c05cad3045032506f88dcdf7b69907b90847d5ade7a340d\": rpc error: code = NotFound desc = could not find container \"ee706cccfcba4bfb4c05cad3045032506f88dcdf7b69907b90847d5ade7a340d\": container with ID starting with ee706cccfcba4bfb4c05cad3045032506f88dcdf7b69907b90847d5ade7a340d not found: ID does not exist"
Oct 30 01:21:47.375935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.375908    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf"]
Oct 30 01:21:47.379606 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.379586    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1ae36-676b8d9d9f-tttlf"]
Oct 30 01:21:47.406503 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.406478    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxk9q\" (UniqueName: \"kubernetes.io/projected/c3dec5ee-7aba-4409-b10d-bed8d78c1df3-kube-api-access-zxk9q\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:47.415560 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.415536    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"]
Oct 30 01:21:47.419678 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.419654    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1ae36-58c758f6d9-x9sm6"]
Oct 30 01:21:47.457684 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.457464    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m"]
Oct 30 01:21:47.465388 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:47.465353    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1ae36-d65d966b6-zkv8m"]
Oct 30 01:21:48.263253 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:48.263182    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57a0acc3-77a4-4ad2-978e-08280840d903" path="/var/lib/kubelet/pods/57a0acc3-77a4-4ad2-978e-08280840d903/volumes"
Oct 30 01:21:48.263668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:48.263654    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75cc8389-4904-44b2-8c92-cf3debf351f4" path="/var/lib/kubelet/pods/75cc8389-4904-44b2-8c92-cf3debf351f4/volumes"
Oct 30 01:21:48.264074 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:48.264061    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3dec5ee-7aba-4409-b10d-bed8d78c1df3" path="/var/lib/kubelet/pods/c3dec5ee-7aba-4409-b10d-bed8d78c1df3/volumes"
Oct 30 01:21:49.928220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:49.928184    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"]
Oct 30 01:21:49.928698 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:49.928446    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq" podUID="ef8abb9d-eff6-478e-9801-c37fd4ef0b7f" containerName="maas-api" containerID="cri-o://81fbfdfda6083ffa89e4e812aa1041b549094d4c2433102be60ac5535db9dc27" gracePeriod=30
Oct 30 01:21:50.295165 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.291686    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"
Oct 30 01:21:50.348293 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.348256    6099 generic.go:356] "Generic (PLEG): container finished" podID="ef8abb9d-eff6-478e-9801-c37fd4ef0b7f" containerID="81fbfdfda6083ffa89e4e812aa1041b549094d4c2433102be60ac5535db9dc27" exitCode=0
Oct 30 01:21:50.348496 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.348420    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq" event={"ID":"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f","Type":"ContainerDied","Data":"81fbfdfda6083ffa89e4e812aa1041b549094d4c2433102be60ac5535db9dc27"}
Oct 30 01:21:50.348496 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.348457    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq" event={"ID":"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f","Type":"ContainerDied","Data":"123a31df02b5f5ed9faf3bc5c6547cbe40dfcc9c111819792830ccd4b05352fe"}
Oct 30 01:21:50.348496 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.348478    6099 scope.go:122] "RemoveContainer" containerID="81fbfdfda6083ffa89e4e812aa1041b549094d4c2433102be60ac5535db9dc27"
Oct 30 01:21:50.348694 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.348583    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"
Oct 30 01:21:50.377888 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.377815    6099 scope.go:122] "RemoveContainer" containerID="81fbfdfda6083ffa89e4e812aa1041b549094d4c2433102be60ac5535db9dc27"
Oct 30 01:21:50.378422 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:21:50.378276    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81fbfdfda6083ffa89e4e812aa1041b549094d4c2433102be60ac5535db9dc27\": container with ID starting with 81fbfdfda6083ffa89e4e812aa1041b549094d4c2433102be60ac5535db9dc27 not found: ID does not exist" containerID="81fbfdfda6083ffa89e4e812aa1041b549094d4c2433102be60ac5535db9dc27"
Oct 30 01:21:50.378422 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.378328    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81fbfdfda6083ffa89e4e812aa1041b549094d4c2433102be60ac5535db9dc27"} err="failed to get container status \"81fbfdfda6083ffa89e4e812aa1041b549094d4c2433102be60ac5535db9dc27\": rpc error: code = NotFound desc = could not find container \"81fbfdfda6083ffa89e4e812aa1041b549094d4c2433102be60ac5535db9dc27\": container with ID starting with 81fbfdfda6083ffa89e4e812aa1041b549094d4c2433102be60ac5535db9dc27 not found: ID does not exist"
Oct 30 01:21:50.416729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.416473    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-metrics-certs\") pod \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\" (UID: \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\") "
Oct 30 01:21:50.416729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.416526    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-maas-api-tls\") pod \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\" (UID: \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\") "
Oct 30 01:21:50.416729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.416577    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-kube-api-access-7ms2z\" (UniqueName: \"kubernetes.io/projected/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-kube-api-access-7ms2z\") pod \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\" (UID: \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\") "
Oct 30 01:21:50.416729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.416640    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-openshift-service-ca\") pod \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\" (UID: \"ef8abb9d-eff6-478e-9801-c37fd4ef0b7f\") "
Oct 30 01:21:50.490840 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.422310    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-openshift-service-ca" pod "ef8abb9d-eff6-478e-9801-c37fd4ef0b7f" (UID: "ef8abb9d-eff6-478e-9801-c37fd4ef0b7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:50.490840 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.449367    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-kube-api-access-7ms2z" pod "ef8abb9d-eff6-478e-9801-c37fd4ef0b7f" (UID: "ef8abb9d-eff6-478e-9801-c37fd4ef0b7f"). InnerVolumeSpecName "kube-api-access-7ms2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:50.542727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.521606    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-maas-api-tls" pod "ef8abb9d-eff6-478e-9801-c37fd4ef0b7f" (UID: "ef8abb9d-eff6-478e-9801-c37fd4ef0b7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:50.542727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.528583    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:50.542727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.528612    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:50.542727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.528626    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ms2z\" (UniqueName: \"kubernetes.io/projected/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-kube-api-access-7ms2z\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:50.542727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.530942    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-metrics-certs" pod "ef8abb9d-eff6-478e-9801-c37fd4ef0b7f" (UID: "ef8abb9d-eff6-478e-9801-c37fd4ef0b7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:50.629963 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.629931    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:50.694971 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.694938    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"]
Oct 30 01:21:50.702072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:50.702047    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-33e759-845857bfdd-grdjq"]
Oct 30 01:21:52.263940 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:52.263909    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef8abb9d-eff6-478e-9801-c37fd4ef0b7f" path="/var/lib/kubelet/pods/ef8abb9d-eff6-478e-9801-c37fd4ef0b7f/volumes"
Oct 30 01:21:52.726292 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:52.723076    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"]
Oct 30 01:21:52.726292 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:52.723368    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9" podUID="87d1ee54-2797-4936-a3a3-44624f7c15c8" containerName="istio-proxy" containerID="cri-o://4d32b59f61aae37df017bebcaf999f2e90c2ba1f9cc84cd34cbd121a7aa756e8" gracePeriod=30
Oct 30 01:21:53.195269 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.195228    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2"]
Oct 30 01:21:53.195972 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.195955    6099 cpu_manager.go:414] "containerMap: removing container" podUID="21a3cb0b-00bf-4b7b-b4cb-976310413ee1" containerName="istio-proxy"
Oct 30 01:21:53.196034 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.195973    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a3cb0b-00bf-4b7b-b4cb-976310413ee1" containerName="istio-proxy"
Oct 30 01:21:53.196034 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.195987    6099 cpu_manager.go:414] "containerMap: removing container" podUID="21a3cb0b-00bf-4b7b-b4cb-976310413ee1" containerName="istio-proxy"
Oct 30 01:21:53.196034 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.195992    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a3cb0b-00bf-4b7b-b4cb-976310413ee1" containerName="istio-proxy"
Oct 30 01:21:53.196034 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196003    6099 cpu_manager.go:414] "containerMap: removing container" podUID="caf7a449-3657-4b2e-aaa6-2cca317f241f" containerName="istio-proxy"
Oct 30 01:21:53.196034 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196009    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf7a449-3657-4b2e-aaa6-2cca317f241f" containerName="istio-proxy"
Oct 30 01:21:53.196034 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196019    6099 cpu_manager.go:414] "containerMap: removing container" podUID="28560cb7-7b8c-46ba-bf1a-afef3f320722" containerName="payload-processing"
Oct 30 01:21:53.196034 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196024    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28560cb7-7b8c-46ba-bf1a-afef3f320722" containerName="payload-processing"
Oct 30 01:21:53.196034 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196033    6099 cpu_manager.go:414] "containerMap: removing container" podUID="75cc8389-4904-44b2-8c92-cf3debf351f4" containerName="maas-api"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196038    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75cc8389-4904-44b2-8c92-cf3debf351f4" containerName="maas-api"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196049    6099 cpu_manager.go:414] "containerMap: removing container" podUID="caf7a449-3657-4b2e-aaa6-2cca317f241f" containerName="istio-proxy"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196054    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf7a449-3657-4b2e-aaa6-2cca317f241f" containerName="istio-proxy"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196062    6099 cpu_manager.go:414] "containerMap: removing container" podUID="c3dec5ee-7aba-4409-b10d-bed8d78c1df3" containerName="payload-pre-processing"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196067    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3dec5ee-7aba-4409-b10d-bed8d78c1df3" containerName="payload-pre-processing"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196084    6099 cpu_manager.go:414] "containerMap: removing container" podUID="1fc8dc61-d744-4c4b-89c5-4122ca0edd22" containerName="revoke-keys"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196089    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fc8dc61-d744-4c4b-89c5-4122ca0edd22" containerName="revoke-keys"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196098    6099 cpu_manager.go:414] "containerMap: removing container" podUID="1bf54661-aa8e-4d60-9981-16006d11e41a" containerName="payload-pre-processing"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196102    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf54661-aa8e-4d60-9981-16006d11e41a" containerName="payload-pre-processing"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196110    6099 cpu_manager.go:414] "containerMap: removing container" podUID="b2e137cc-9cf8-4212-a7cc-0759867fd7ea" containerName="payload-processing"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196114    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e137cc-9cf8-4212-a7cc-0759867fd7ea" containerName="payload-processing"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196121    6099 cpu_manager.go:414] "containerMap: removing container" podUID="57a0acc3-77a4-4ad2-978e-08280840d903" containerName="payload-processing"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196125    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a0acc3-77a4-4ad2-978e-08280840d903" containerName="payload-processing"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196133    6099 cpu_manager.go:414] "containerMap: removing container" podUID="1c5c9922-e50a-47a1-8b74-021582765920" containerName="istio-proxy"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196137    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5c9922-e50a-47a1-8b74-021582765920" containerName="istio-proxy"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196145    6099 cpu_manager.go:414] "containerMap: removing container" podUID="41fd810f-4559-492b-8038-7cd547121618" containerName="revoke-keys"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196149    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41fd810f-4559-492b-8038-7cd547121618" containerName="revoke-keys"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196162    6099 cpu_manager.go:414] "containerMap: removing container" podUID="7c1533e0-8ff9-4108-8688-714e05957447" containerName="istio-proxy"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196166    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1533e0-8ff9-4108-8688-714e05957447" containerName="istio-proxy"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196175    6099 cpu_manager.go:414] "containerMap: removing container" podUID="605336de-7c5e-42ed-bc77-2f6d56507518" containerName="payload-pre-processing"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196180    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="605336de-7c5e-42ed-bc77-2f6d56507518" containerName="payload-pre-processing"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196198    6099 cpu_manager.go:414] "containerMap: removing container" podUID="ef8abb9d-eff6-478e-9801-c37fd4ef0b7f" containerName="maas-api"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196202    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef8abb9d-eff6-478e-9801-c37fd4ef0b7f" containerName="maas-api"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196293    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a3cb0b-00bf-4b7b-b4cb-976310413ee1" containerName="istio-proxy"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196301    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf7a449-3657-4b2e-aaa6-2cca317f241f" containerName="istio-proxy"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196310    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="28560cb7-7b8c-46ba-bf1a-afef3f320722" containerName="payload-processing"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196317    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="75cc8389-4904-44b2-8c92-cf3debf351f4" containerName="maas-api"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196324    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf7a449-3657-4b2e-aaa6-2cca317f241f" containerName="istio-proxy"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196331    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3dec5ee-7aba-4409-b10d-bed8d78c1df3" containerName="payload-pre-processing"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196338    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fc8dc61-d744-4c4b-89c5-4122ca0edd22" containerName="revoke-keys"
Oct 30 01:21:53.196335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196346    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf54661-aa8e-4d60-9981-16006d11e41a" containerName="payload-pre-processing"
Oct 30 01:21:53.197347 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196351    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e137cc-9cf8-4212-a7cc-0759867fd7ea" containerName="payload-processing"
Oct 30 01:21:53.197347 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196356    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a0acc3-77a4-4ad2-978e-08280840d903" containerName="payload-processing"
Oct 30 01:21:53.197347 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196363    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5c9922-e50a-47a1-8b74-021582765920" containerName="istio-proxy"
Oct 30 01:21:53.197347 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196370    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="41fd810f-4559-492b-8038-7cd547121618" containerName="revoke-keys"
Oct 30 01:21:53.197347 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196376    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1533e0-8ff9-4108-8688-714e05957447" containerName="istio-proxy"
Oct 30 01:21:53.197347 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196382    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="605336de-7c5e-42ed-bc77-2f6d56507518" containerName="payload-pre-processing"
Oct 30 01:21:53.197347 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.196390    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef8abb9d-eff6-478e-9801-c37fd4ef0b7f" containerName="maas-api"
Oct 30 01:21:53.201637 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.201618    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2"
Oct 30 01:21:53.218169 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.218146    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:21:53.218307 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.218293    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:21:53.221031 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.221002    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2"]
Oct 30 01:21:53.386834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.386803    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87843f3-153e-4fc8-9771-316dfd0d2668-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2\" (UID: \"e87843f3-153e-4fc8-9771-316dfd0d2668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2"
Oct 30 01:21:53.387369 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.387244    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87843f3-153e-4fc8-9771-316dfd0d2668-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2\" (UID: \"e87843f3-153e-4fc8-9771-316dfd0d2668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2"
Oct 30 01:21:53.387453 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.387389    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtkm7\" (UniqueName: \"kubernetes.io/projected/e87843f3-153e-4fc8-9771-316dfd0d2668-kube-api-access-xtkm7\") pod \"maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2\" (UID: \"e87843f3-153e-4fc8-9771-316dfd0d2668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2"
Oct 30 01:21:53.489514 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.489417    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87843f3-153e-4fc8-9771-316dfd0d2668-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2\" (UID: \"e87843f3-153e-4fc8-9771-316dfd0d2668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2"
Oct 30 01:21:53.489741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.489663    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87843f3-153e-4fc8-9771-316dfd0d2668-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2\" (UID: \"e87843f3-153e-4fc8-9771-316dfd0d2668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2"
Oct 30 01:21:53.489827 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.489770    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtkm7\" (UniqueName: \"kubernetes.io/projected/e87843f3-153e-4fc8-9771-316dfd0d2668-kube-api-access-xtkm7\") pod \"maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2\" (UID: \"e87843f3-153e-4fc8-9771-316dfd0d2668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2"
Oct 30 01:21:53.490200 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.490180    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87843f3-153e-4fc8-9771-316dfd0d2668-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2\" (UID: \"e87843f3-153e-4fc8-9771-316dfd0d2668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2"
Oct 30 01:21:53.492018 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.491995    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87843f3-153e-4fc8-9771-316dfd0d2668-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2\" (UID: \"e87843f3-153e-4fc8-9771-316dfd0d2668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2"
Oct 30 01:21:53.514035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.514012    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtkm7\" (UniqueName: \"kubernetes.io/projected/e87843f3-153e-4fc8-9771-316dfd0d2668-kube-api-access-xtkm7\") pod \"maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2\" (UID: \"e87843f3-153e-4fc8-9771-316dfd0d2668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2"
Oct 30 01:21:53.517157 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.517138    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2"
Oct 30 01:21:53.826630 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:53.826598    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2"]
Oct 30 01:21:53.827412 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:21:53.827387    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87843f3_153e_4fc8_9771_316dfd0d2668.slice/crio-461070a66aa1c85467f09d2cd0d46e38b445a886589768d401e94237a7d8c36f WatchSource:0}: Error finding container 461070a66aa1c85467f09d2cd0d46e38b445a886589768d401e94237a7d8c36f: Status 404 returned error can't find the container with id 461070a66aa1c85467f09d2cd0d46e38b445a886589768d401e94237a7d8c36f
Oct 30 01:21:54.380825 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:54.380733    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2" event={"ID":"e87843f3-153e-4fc8-9771-316dfd0d2668","Type":"ContainerStarted","Data":"65aa649fd58c23d082c789c35989fd407fc73c59d199f3ec9efcddb9a228024e"}
Oct 30 01:21:54.380825 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:54.380797    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2" event={"ID":"e87843f3-153e-4fc8-9771-316dfd0d2668","Type":"ContainerStarted","Data":"461070a66aa1c85467f09d2cd0d46e38b445a886589768d401e94237a7d8c36f"}
Oct 30 01:21:54.475148 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:54.475090    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2" podStartSLOduration=1.475073855 podStartE2EDuration="1.475073855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:54.471443786 +0000 UTC m=+2808.328351542" watchObservedRunningTime="2026-10-30 01:21:54.475073855 +0000 UTC m=+2808.331981604"
Oct 30 01:21:55.104863 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:55.104830    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:21:55.389477 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:55.389395    6099 generic.go:356] "Generic (PLEG): container finished" podID="e87843f3-153e-4fc8-9771-316dfd0d2668" containerID="65aa649fd58c23d082c789c35989fd407fc73c59d199f3ec9efcddb9a228024e" exitCode=0
Oct 30 01:21:55.389477 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:55.389443    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2" event={"ID":"e87843f3-153e-4fc8-9771-316dfd0d2668","Type":"ContainerDied","Data":"65aa649fd58c23d082c789c35989fd407fc73c59d199f3ec9efcddb9a228024e"}
Oct 30 01:21:56.696491 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:56.696464    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2"
Oct 30 01:21:56.765456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:56.765368    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e87843f3-153e-4fc8-9771-316dfd0d2668-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87843f3-153e-4fc8-9771-316dfd0d2668-maas-api-xks-ca\") pod \"e87843f3-153e-4fc8-9771-316dfd0d2668\" (UID: \"e87843f3-153e-4fc8-9771-316dfd0d2668\") "
Oct 30 01:21:56.765456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:56.765436    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e87843f3-153e-4fc8-9771-316dfd0d2668-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87843f3-153e-4fc8-9771-316dfd0d2668-maas-api-openshift-ca\") pod \"e87843f3-153e-4fc8-9771-316dfd0d2668\" (UID: \"e87843f3-153e-4fc8-9771-316dfd0d2668\") "
Oct 30 01:21:56.765721 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:56.765613    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e87843f3-153e-4fc8-9771-316dfd0d2668-kube-api-access-xtkm7\" (UniqueName: \"kubernetes.io/projected/e87843f3-153e-4fc8-9771-316dfd0d2668-kube-api-access-xtkm7\") pod \"e87843f3-153e-4fc8-9771-316dfd0d2668\" (UID: \"e87843f3-153e-4fc8-9771-316dfd0d2668\") "
Oct 30 01:21:56.765957 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:56.765926    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87843f3-153e-4fc8-9771-316dfd0d2668-maas-api-openshift-ca" pod "e87843f3-153e-4fc8-9771-316dfd0d2668" (UID: "e87843f3-153e-4fc8-9771-316dfd0d2668"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:56.766961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:56.766933    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87843f3-153e-4fc8-9771-316dfd0d2668-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:56.767809 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:56.767781    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e87843f3-153e-4fc8-9771-316dfd0d2668-maas-api-xks-ca" pod "e87843f3-153e-4fc8-9771-316dfd0d2668" (UID: "e87843f3-153e-4fc8-9771-316dfd0d2668"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:56.767809 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:56.768276    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87843f3-153e-4fc8-9771-316dfd0d2668-kube-api-access-xtkm7" pod "e87843f3-153e-4fc8-9771-316dfd0d2668" (UID: "e87843f3-153e-4fc8-9771-316dfd0d2668"). InnerVolumeSpecName "kube-api-access-xtkm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:56.867938 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:56.867906    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87843f3-153e-4fc8-9771-316dfd0d2668-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:56.867938 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:56.867936    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtkm7\" (UniqueName: \"kubernetes.io/projected/e87843f3-153e-4fc8-9771-316dfd0d2668-kube-api-access-xtkm7\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:57.450541 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:57.450514    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2"
Oct 30 01:21:57.450739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:57.450560    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2" event={"ID":"e87843f3-153e-4fc8-9771-316dfd0d2668","Type":"ContainerDied","Data":"461070a66aa1c85467f09d2cd0d46e38b445a886589768d401e94237a7d8c36f"}
Oct 30 01:21:57.450739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:57.450602    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="461070a66aa1c85467f09d2cd0d46e38b445a886589768d401e94237a7d8c36f"
Oct 30 01:21:58.028668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.028640    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:21:58.104126 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.092317    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-token\" (UniqueName: \"kubernetes.io/projected/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-token\") pod \"87d1ee54-2797-4936-a3a3-44624f7c15c8\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") "
Oct 30 01:21:58.104126 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.092387    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-workload-certs\") pod \"87d1ee54-2797-4936-a3a3-44624f7c15c8\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") "
Oct 30 01:21:58.104126 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.092493    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87d1ee54-2797-4936-a3a3-44624f7c15c8-kube-api-access-g66jc\" (UniqueName: \"kubernetes.io/projected/87d1ee54-2797-4936-a3a3-44624f7c15c8-kube-api-access-g66jc\") pod \"87d1ee54-2797-4936-a3a3-44624f7c15c8\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") "
Oct 30 01:21:58.104126 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.092581    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-data\") pod \"87d1ee54-2797-4936-a3a3-44624f7c15c8\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") "
Oct 30 01:21:58.104126 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.092652    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-podinfo\") pod \"87d1ee54-2797-4936-a3a3-44624f7c15c8\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") "
Oct 30 01:21:58.104126 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.092755    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-credential-socket\") pod \"87d1ee54-2797-4936-a3a3-44624f7c15c8\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") "
Oct 30 01:21:58.104126 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.092795    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-envoy\") pod \"87d1ee54-2797-4936-a3a3-44624f7c15c8\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") "
Oct 30 01:21:58.104126 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.092940    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-workload-socket\") pod \"87d1ee54-2797-4936-a3a3-44624f7c15c8\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") "
Oct 30 01:21:58.104126 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.092965    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87d1ee54-2797-4936-a3a3-44624f7c15c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87d1ee54-2797-4936-a3a3-44624f7c15c8-istiod-ca-cert\") pod \"87d1ee54-2797-4936-a3a3-44624f7c15c8\" (UID: \"87d1ee54-2797-4936-a3a3-44624f7c15c8\") "
Oct 30 01:21:58.104126 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.101949    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-credential-socket" pod "87d1ee54-2797-4936-a3a3-44624f7c15c8" (UID: "87d1ee54-2797-4936-a3a3-44624f7c15c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:58.104126 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.104010    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-data" pod "87d1ee54-2797-4936-a3a3-44624f7c15c8" (UID: "87d1ee54-2797-4936-a3a3-44624f7c15c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:58.104875 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.104191    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-workload-certs" pod "87d1ee54-2797-4936-a3a3-44624f7c15c8" (UID: "87d1ee54-2797-4936-a3a3-44624f7c15c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:58.104875 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.104366    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-workload-socket" pod "87d1ee54-2797-4936-a3a3-44624f7c15c8" (UID: "87d1ee54-2797-4936-a3a3-44624f7c15c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:58.106142 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.106117    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d1ee54-2797-4936-a3a3-44624f7c15c8-istiod-ca-cert" pod "87d1ee54-2797-4936-a3a3-44624f7c15c8" (UID: "87d1ee54-2797-4936-a3a3-44624f7c15c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:58.106899 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.106838    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-token" pod "87d1ee54-2797-4936-a3a3-44624f7c15c8" (UID: "87d1ee54-2797-4936-a3a3-44624f7c15c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:58.107071 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.107042    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-podinfo" pod "87d1ee54-2797-4936-a3a3-44624f7c15c8" (UID: "87d1ee54-2797-4936-a3a3-44624f7c15c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:21:58.114878 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.114842    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-envoy" pod "87d1ee54-2797-4936-a3a3-44624f7c15c8" (UID: "87d1ee54-2797-4936-a3a3-44624f7c15c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:58.125765 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.125718    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d1ee54-2797-4936-a3a3-44624f7c15c8-kube-api-access-g66jc" pod "87d1ee54-2797-4936-a3a3-44624f7c15c8" (UID: "87d1ee54-2797-4936-a3a3-44624f7c15c8"). InnerVolumeSpecName "kube-api-access-g66jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:58.195668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.195634    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:58.195668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.195666    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:58.195668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.195675    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:58.195942 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.195684    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:58.195942 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.195693    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:58.195942 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.195701    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87d1ee54-2797-4936-a3a3-44624f7c15c8-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:58.195942 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.195726    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87d1ee54-2797-4936-a3a3-44624f7c15c8-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:58.195942 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.195736    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87d1ee54-2797-4936-a3a3-44624f7c15c8-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:58.195942 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.195743    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g66jc\" (UniqueName: \"kubernetes.io/projected/87d1ee54-2797-4936-a3a3-44624f7c15c8-kube-api-access-g66jc\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:21:58.460254 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.460220    6099 generic.go:356] "Generic (PLEG): container finished" podID="87d1ee54-2797-4936-a3a3-44624f7c15c8" containerID="4d32b59f61aae37df017bebcaf999f2e90c2ba1f9cc84cd34cbd121a7aa756e8" exitCode=0
Oct 30 01:21:58.460460 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.460275    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"
Oct 30 01:21:58.460460 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.460297    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9" event={"ID":"87d1ee54-2797-4936-a3a3-44624f7c15c8","Type":"ContainerDied","Data":"4d32b59f61aae37df017bebcaf999f2e90c2ba1f9cc84cd34cbd121a7aa756e8"}
Oct 30 01:21:58.460460 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.460355    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9" event={"ID":"87d1ee54-2797-4936-a3a3-44624f7c15c8","Type":"ContainerDied","Data":"b8e363766b89083a1325120df0e579b5317b70068546155a987f24ea7cf0f708"}
Oct 30 01:21:58.460460 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.460387    6099 scope.go:122] "RemoveContainer" containerID="4d32b59f61aae37df017bebcaf999f2e90c2ba1f9cc84cd34cbd121a7aa756e8"
Oct 30 01:21:58.473693 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.473672    6099 scope.go:122] "RemoveContainer" containerID="3bf85cbe38b97a8dec59f3de6329addc6f202683d89930fd17bdf53370a74835"
Oct 30 01:21:58.483689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.483665    6099 scope.go:122] "RemoveContainer" containerID="4d32b59f61aae37df017bebcaf999f2e90c2ba1f9cc84cd34cbd121a7aa756e8"
Oct 30 01:21:58.484006 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:21:58.483986    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d32b59f61aae37df017bebcaf999f2e90c2ba1f9cc84cd34cbd121a7aa756e8\": container with ID starting with 4d32b59f61aae37df017bebcaf999f2e90c2ba1f9cc84cd34cbd121a7aa756e8 not found: ID does not exist" containerID="4d32b59f61aae37df017bebcaf999f2e90c2ba1f9cc84cd34cbd121a7aa756e8"
Oct 30 01:21:58.484063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.484014    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d32b59f61aae37df017bebcaf999f2e90c2ba1f9cc84cd34cbd121a7aa756e8"} err="failed to get container status \"4d32b59f61aae37df017bebcaf999f2e90c2ba1f9cc84cd34cbd121a7aa756e8\": rpc error: code = NotFound desc = could not find container \"4d32b59f61aae37df017bebcaf999f2e90c2ba1f9cc84cd34cbd121a7aa756e8\": container with ID starting with 4d32b59f61aae37df017bebcaf999f2e90c2ba1f9cc84cd34cbd121a7aa756e8 not found: ID does not exist"
Oct 30 01:21:58.484063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.484032    6099 scope.go:122] "RemoveContainer" containerID="3bf85cbe38b97a8dec59f3de6329addc6f202683d89930fd17bdf53370a74835"
Oct 30 01:21:58.484254 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:21:58.484239    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf85cbe38b97a8dec59f3de6329addc6f202683d89930fd17bdf53370a74835\": container with ID starting with 3bf85cbe38b97a8dec59f3de6329addc6f202683d89930fd17bdf53370a74835 not found: ID does not exist" containerID="3bf85cbe38b97a8dec59f3de6329addc6f202683d89930fd17bdf53370a74835"
Oct 30 01:21:58.484306 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.484257    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf85cbe38b97a8dec59f3de6329addc6f202683d89930fd17bdf53370a74835"} err="failed to get container status \"3bf85cbe38b97a8dec59f3de6329addc6f202683d89930fd17bdf53370a74835\": rpc error: code = NotFound desc = could not find container \"3bf85cbe38b97a8dec59f3de6329addc6f202683d89930fd17bdf53370a74835\": container with ID starting with 3bf85cbe38b97a8dec59f3de6329addc6f202683d89930fd17bdf53370a74835 not found: ID does not exist"
Oct 30 01:21:58.512024 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.511997    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"]
Oct 30 01:21:58.525812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.525784    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-481249-openshift-default-6c77499cf5-phgd9"]
Oct 30 01:21:58.825121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.825087    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"]
Oct 30 01:21:58.825383 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:58.825345    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb" podUID="673e2d41-cb67-4956-bb96-959aa37fb924" containerName="istio-proxy" containerID="cri-o://45c62a6b60da5713aa75144e46981591c3b1806df2684d95cbeb42d1711b868f" gracePeriod=30
Oct 30 01:21:59.575684 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.575651    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8"]
Oct 30 01:21:59.576378 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.576359    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e87843f3-153e-4fc8-9771-316dfd0d2668" containerName="revoke-keys"
Oct 30 01:21:59.576378 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.576378    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87843f3-153e-4fc8-9771-316dfd0d2668" containerName="revoke-keys"
Oct 30 01:21:59.576505 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.576393    6099 cpu_manager.go:414] "containerMap: removing container" podUID="87d1ee54-2797-4936-a3a3-44624f7c15c8" containerName="istio-proxy"
Oct 30 01:21:59.576505 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.576399    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87d1ee54-2797-4936-a3a3-44624f7c15c8" containerName="istio-proxy"
Oct 30 01:21:59.576505 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.576412    6099 cpu_manager.go:414] "containerMap: removing container" podUID="87d1ee54-2797-4936-a3a3-44624f7c15c8" containerName="istio-proxy"
Oct 30 01:21:59.576505 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.576417    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87d1ee54-2797-4936-a3a3-44624f7c15c8" containerName="istio-proxy"
Oct 30 01:21:59.577224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.576540    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a3cb0b-00bf-4b7b-b4cb-976310413ee1" containerName="istio-proxy"
Oct 30 01:21:59.577224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.576557    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e87843f3-153e-4fc8-9771-316dfd0d2668" containerName="revoke-keys"
Oct 30 01:21:59.577224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.576563    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="87d1ee54-2797-4936-a3a3-44624f7c15c8" containerName="istio-proxy"
Oct 30 01:21:59.581747 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.581729    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8"
Oct 30 01:21:59.599460 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.599435    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8"]
Oct 30 01:21:59.606670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.606648    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:21:59.606817 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.606795    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:21:59.661519 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.661488    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdvjv\" (UniqueName: \"kubernetes.io/projected/9af17237-0f9e-435c-b453-ced511ca7425-kube-api-access-kdvjv\") pod \"maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8\" (UID: \"9af17237-0f9e-435c-b453-ced511ca7425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8"
Oct 30 01:21:59.661680 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.661634    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af17237-0f9e-435c-b453-ced511ca7425-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8\" (UID: \"9af17237-0f9e-435c-b453-ced511ca7425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8"
Oct 30 01:21:59.661822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.661697    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af17237-0f9e-435c-b453-ced511ca7425-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8\" (UID: \"9af17237-0f9e-435c-b453-ced511ca7425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8"
Oct 30 01:21:59.763395 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.763357    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdvjv\" (UniqueName: \"kubernetes.io/projected/9af17237-0f9e-435c-b453-ced511ca7425-kube-api-access-kdvjv\") pod \"maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8\" (UID: \"9af17237-0f9e-435c-b453-ced511ca7425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8"
Oct 30 01:21:59.763560 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.763467    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af17237-0f9e-435c-b453-ced511ca7425-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8\" (UID: \"9af17237-0f9e-435c-b453-ced511ca7425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8"
Oct 30 01:21:59.763560 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.763522    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af17237-0f9e-435c-b453-ced511ca7425-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8\" (UID: \"9af17237-0f9e-435c-b453-ced511ca7425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8"
Oct 30 01:21:59.764130 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.764110    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af17237-0f9e-435c-b453-ced511ca7425-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8\" (UID: \"9af17237-0f9e-435c-b453-ced511ca7425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8"
Oct 30 01:21:59.765921 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.765898    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af17237-0f9e-435c-b453-ced511ca7425-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8\" (UID: \"9af17237-0f9e-435c-b453-ced511ca7425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8"
Oct 30 01:21:59.785797 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.785772    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdvjv\" (UniqueName: \"kubernetes.io/projected/9af17237-0f9e-435c-b453-ced511ca7425-kube-api-access-kdvjv\") pod \"maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8\" (UID: \"9af17237-0f9e-435c-b453-ced511ca7425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8"
Oct 30 01:21:59.896089 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:21:59.896018    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8"
Oct 30 01:22:00.202689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:00.202658    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8"]
Oct 30 01:22:00.204647 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:22:00.204615    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af17237_0f9e_435c_b453_ced511ca7425.slice/crio-34b8d5ba1cda4b3b4e9e52aa144ab9d9bd07592b77f67fc5690c55432e483e36 WatchSource:0}: Error finding container 34b8d5ba1cda4b3b4e9e52aa144ab9d9bd07592b77f67fc5690c55432e483e36: Status 404 returned error can't find the container with id 34b8d5ba1cda4b3b4e9e52aa144ab9d9bd07592b77f67fc5690c55432e483e36
Oct 30 01:22:00.263026 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:00.263003    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87d1ee54-2797-4936-a3a3-44624f7c15c8" path="/var/lib/kubelet/pods/87d1ee54-2797-4936-a3a3-44624f7c15c8/volumes"
Oct 30 01:22:00.488442 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:00.487439    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8" event={"ID":"9af17237-0f9e-435c-b453-ced511ca7425","Type":"ContainerStarted","Data":"34b8d5ba1cda4b3b4e9e52aa144ab9d9bd07592b77f67fc5690c55432e483e36"}
Oct 30 01:22:01.496217 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:01.496133    6099 generic.go:356] "Generic (PLEG): container finished" podID="9af17237-0f9e-435c-b453-ced511ca7425" containerID="ec53ff181874d2d99f502855dec3aa517c63bb474090e968581289dcb40e2fa0" exitCode=0
Oct 30 01:22:01.496217 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:01.496192    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8" event={"ID":"9af17237-0f9e-435c-b453-ced511ca7425","Type":"ContainerDied","Data":"ec53ff181874d2d99f502855dec3aa517c63bb474090e968581289dcb40e2fa0"}
Oct 30 01:22:02.765753 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:02.765681    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8"
Oct 30 01:22:02.862701 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:02.862666    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9af17237-0f9e-435c-b453-ced511ca7425-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af17237-0f9e-435c-b453-ced511ca7425-maas-api-xks-ca\") pod \"9af17237-0f9e-435c-b453-ced511ca7425\" (UID: \"9af17237-0f9e-435c-b453-ced511ca7425\") "
Oct 30 01:22:02.863197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:02.863168    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9af17237-0f9e-435c-b453-ced511ca7425-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af17237-0f9e-435c-b453-ced511ca7425-maas-api-openshift-ca\") pod \"9af17237-0f9e-435c-b453-ced511ca7425\" (UID: \"9af17237-0f9e-435c-b453-ced511ca7425\") "
Oct 30 01:22:02.863370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:02.863354    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af17237-0f9e-435c-b453-ced511ca7425-kube-api-access-kdvjv\" (UniqueName: \"kubernetes.io/projected/9af17237-0f9e-435c-b453-ced511ca7425-kube-api-access-kdvjv\") pod \"9af17237-0f9e-435c-b453-ced511ca7425\" (UID: \"9af17237-0f9e-435c-b453-ced511ca7425\") "
Oct 30 01:22:02.895311 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:02.893442    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9af17237-0f9e-435c-b453-ced511ca7425-maas-api-xks-ca" pod "9af17237-0f9e-435c-b453-ced511ca7425" (UID: "9af17237-0f9e-435c-b453-ced511ca7425"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:02.906728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:02.895681    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af17237-0f9e-435c-b453-ced511ca7425-maas-api-openshift-ca" pod "9af17237-0f9e-435c-b453-ced511ca7425" (UID: "9af17237-0f9e-435c-b453-ced511ca7425"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:02.944744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:02.943431    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af17237-0f9e-435c-b453-ced511ca7425-kube-api-access-kdvjv" pod "9af17237-0f9e-435c-b453-ced511ca7425" (UID: "9af17237-0f9e-435c-b453-ced511ca7425"). InnerVolumeSpecName "kube-api-access-kdvjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:02.972748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:02.971908    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af17237-0f9e-435c-b453-ced511ca7425-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:02.972748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:02.971943    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af17237-0f9e-435c-b453-ced511ca7425-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:02.972748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:02.971973    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdvjv\" (UniqueName: \"kubernetes.io/projected/9af17237-0f9e-435c-b453-ced511ca7425-kube-api-access-kdvjv\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:03.512971 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:03.512946    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8"
Oct 30 01:22:03.513192 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:03.512941    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8" event={"ID":"9af17237-0f9e-435c-b453-ced511ca7425","Type":"ContainerDied","Data":"34b8d5ba1cda4b3b4e9e52aa144ab9d9bd07592b77f67fc5690c55432e483e36"}
Oct 30 01:22:03.513192 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:03.513065    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34b8d5ba1cda4b3b4e9e52aa144ab9d9bd07592b77f67fc5690c55432e483e36"
Oct 30 01:22:04.124906 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.124877    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:22:04.192852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.189753    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/673e2d41-cb67-4956-bb96-959aa37fb924-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/673e2d41-cb67-4956-bb96-959aa37fb924-istiod-ca-cert\") pod \"673e2d41-cb67-4956-bb96-959aa37fb924\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") "
Oct 30 01:22:04.192852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.189865    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-istio-envoy\") pod \"673e2d41-cb67-4956-bb96-959aa37fb924\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") "
Oct 30 01:22:04.192852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.190056    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-workload-socket\") pod \"673e2d41-cb67-4956-bb96-959aa37fb924\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") "
Oct 30 01:22:04.192852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.190104    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/673e2d41-cb67-4956-bb96-959aa37fb924-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/673e2d41-cb67-4956-bb96-959aa37fb924-istio-podinfo\") pod \"673e2d41-cb67-4956-bb96-959aa37fb924\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") "
Oct 30 01:22:04.192852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.190161    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/673e2d41-cb67-4956-bb96-959aa37fb924-istio-token\" (UniqueName: \"kubernetes.io/projected/673e2d41-cb67-4956-bb96-959aa37fb924-istio-token\") pod \"673e2d41-cb67-4956-bb96-959aa37fb924\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") "
Oct 30 01:22:04.192852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.190242    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-workload-certs\") pod \"673e2d41-cb67-4956-bb96-959aa37fb924\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") "
Oct 30 01:22:04.192852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.190341    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-istio-data\") pod \"673e2d41-cb67-4956-bb96-959aa37fb924\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") "
Oct 30 01:22:04.192852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.190493    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/673e2d41-cb67-4956-bb96-959aa37fb924-kube-api-access-stldw\" (UniqueName: \"kubernetes.io/projected/673e2d41-cb67-4956-bb96-959aa37fb924-kube-api-access-stldw\") pod \"673e2d41-cb67-4956-bb96-959aa37fb924\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") "
Oct 30 01:22:04.192852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.190587    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-credential-socket\") pod \"673e2d41-cb67-4956-bb96-959aa37fb924\" (UID: \"673e2d41-cb67-4956-bb96-959aa37fb924\") "
Oct 30 01:22:04.192852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.191996    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-credential-socket" pod "673e2d41-cb67-4956-bb96-959aa37fb924" (UID: "673e2d41-cb67-4956-bb96-959aa37fb924"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:04.192852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.192449    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/673e2d41-cb67-4956-bb96-959aa37fb924-istiod-ca-cert" pod "673e2d41-cb67-4956-bb96-959aa37fb924" (UID: "673e2d41-cb67-4956-bb96-959aa37fb924"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:04.194552 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.194406    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-workload-certs" pod "673e2d41-cb67-4956-bb96-959aa37fb924" (UID: "673e2d41-cb67-4956-bb96-959aa37fb924"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:04.194552 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.194522    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-istio-data" pod "673e2d41-cb67-4956-bb96-959aa37fb924" (UID: "673e2d41-cb67-4956-bb96-959aa37fb924"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:04.194873 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.194838    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-workload-socket" pod "673e2d41-cb67-4956-bb96-959aa37fb924" (UID: "673e2d41-cb67-4956-bb96-959aa37fb924"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:04.196881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.196847    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/673e2d41-cb67-4956-bb96-959aa37fb924-istio-podinfo" pod "673e2d41-cb67-4956-bb96-959aa37fb924" (UID: "673e2d41-cb67-4956-bb96-959aa37fb924"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:22:04.196980 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.196922    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673e2d41-cb67-4956-bb96-959aa37fb924-kube-api-access-stldw" pod "673e2d41-cb67-4956-bb96-959aa37fb924" (UID: "673e2d41-cb67-4956-bb96-959aa37fb924"). InnerVolumeSpecName "kube-api-access-stldw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:04.203394 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.203363    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673e2d41-cb67-4956-bb96-959aa37fb924-istio-token" pod "673e2d41-cb67-4956-bb96-959aa37fb924" (UID: "673e2d41-cb67-4956-bb96-959aa37fb924"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:04.204505 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.204477    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-istio-envoy" pod "673e2d41-cb67-4956-bb96-959aa37fb924" (UID: "673e2d41-cb67-4956-bb96-959aa37fb924"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:04.293484 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.293450    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/673e2d41-cb67-4956-bb96-959aa37fb924-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:04.293484 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.293478    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:04.293484 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.293492    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:04.293812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.293500    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/673e2d41-cb67-4956-bb96-959aa37fb924-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:04.293812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.293512    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/673e2d41-cb67-4956-bb96-959aa37fb924-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:04.293812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.293525    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:04.293812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.293535    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:04.293812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.293547    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stldw\" (UniqueName: \"kubernetes.io/projected/673e2d41-cb67-4956-bb96-959aa37fb924-kube-api-access-stldw\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:04.293812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.293558    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/673e2d41-cb67-4956-bb96-959aa37fb924-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:04.296159 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.296129    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"]
Oct 30 01:22:04.523817 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.523679    6099 generic.go:356] "Generic (PLEG): container finished" podID="673e2d41-cb67-4956-bb96-959aa37fb924" containerID="45c62a6b60da5713aa75144e46981591c3b1806df2684d95cbeb42d1711b868f" exitCode=0
Oct 30 01:22:04.523817 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.523758    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb" event={"ID":"673e2d41-cb67-4956-bb96-959aa37fb924","Type":"ContainerDied","Data":"45c62a6b60da5713aa75144e46981591c3b1806df2684d95cbeb42d1711b868f"}
Oct 30 01:22:04.523817 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.523774    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"
Oct 30 01:22:04.523817 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.523817    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb" event={"ID":"673e2d41-cb67-4956-bb96-959aa37fb924","Type":"ContainerDied","Data":"43bff4f061f176cbc1742ef7168afafb11cd8f8958168ad84e4b93a194ac1e09"}
Oct 30 01:22:04.524226 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.523841    6099 scope.go:122] "RemoveContainer" containerID="45c62a6b60da5713aa75144e46981591c3b1806df2684d95cbeb42d1711b868f"
Oct 30 01:22:04.537702 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.537679    6099 scope.go:122] "RemoveContainer" containerID="95bd9bbccc2e6311c11db1dc7dcb68c104620dfa0d7824adb9c15f44f4c78976"
Oct 30 01:22:04.547977 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.547959    6099 scope.go:122] "RemoveContainer" containerID="45c62a6b60da5713aa75144e46981591c3b1806df2684d95cbeb42d1711b868f"
Oct 30 01:22:04.548227 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:22:04.548210    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c62a6b60da5713aa75144e46981591c3b1806df2684d95cbeb42d1711b868f\": container with ID starting with 45c62a6b60da5713aa75144e46981591c3b1806df2684d95cbeb42d1711b868f not found: ID does not exist" containerID="45c62a6b60da5713aa75144e46981591c3b1806df2684d95cbeb42d1711b868f"
Oct 30 01:22:04.548275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.548234    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c62a6b60da5713aa75144e46981591c3b1806df2684d95cbeb42d1711b868f"} err="failed to get container status \"45c62a6b60da5713aa75144e46981591c3b1806df2684d95cbeb42d1711b868f\": rpc error: code = NotFound desc = could not find container \"45c62a6b60da5713aa75144e46981591c3b1806df2684d95cbeb42d1711b868f\": container with ID starting with 45c62a6b60da5713aa75144e46981591c3b1806df2684d95cbeb42d1711b868f not found: ID does not exist"
Oct 30 01:22:04.548275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.548249    6099 scope.go:122] "RemoveContainer" containerID="95bd9bbccc2e6311c11db1dc7dcb68c104620dfa0d7824adb9c15f44f4c78976"
Oct 30 01:22:04.548518 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:22:04.548500    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95bd9bbccc2e6311c11db1dc7dcb68c104620dfa0d7824adb9c15f44f4c78976\": container with ID starting with 95bd9bbccc2e6311c11db1dc7dcb68c104620dfa0d7824adb9c15f44f4c78976 not found: ID does not exist" containerID="95bd9bbccc2e6311c11db1dc7dcb68c104620dfa0d7824adb9c15f44f4c78976"
Oct 30 01:22:04.548564 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.548523    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95bd9bbccc2e6311c11db1dc7dcb68c104620dfa0d7824adb9c15f44f4c78976"} err="failed to get container status \"95bd9bbccc2e6311c11db1dc7dcb68c104620dfa0d7824adb9c15f44f4c78976\": rpc error: code = NotFound desc = could not find container \"95bd9bbccc2e6311c11db1dc7dcb68c104620dfa0d7824adb9c15f44f4c78976\": container with ID starting with 95bd9bbccc2e6311c11db1dc7dcb68c104620dfa0d7824adb9c15f44f4c78976 not found: ID does not exist"
Oct 30 01:22:04.575657 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.575632    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"]
Oct 30 01:22:04.582456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:04.582435    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-33e759-openshift-default-845bcbf7fc-57rgb"]
Oct 30 01:22:06.263108 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:06.263076    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="673e2d41-cb67-4956-bb96-959aa37fb924" path="/var/lib/kubelet/pods/673e2d41-cb67-4956-bb96-959aa37fb924/volumes"
Oct 30 01:22:06.433754 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:06.428877    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"]
Oct 30 01:22:06.433754 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:06.429116    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72" podUID="bdf12444-9b7c-4721-900b-44b11df44dd1" containerName="istio-proxy" containerID="cri-o://cc1b96ec9b18656aeac10b69e06acc1438e7595307bb741bb3a5ef4ab8072137" gracePeriod=30
Oct 30 01:22:07.138737 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.137763    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq"]
Oct 30 01:22:07.138977 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.138834    6099 cpu_manager.go:414] "containerMap: removing container" podUID="673e2d41-cb67-4956-bb96-959aa37fb924" containerName="istio-proxy"
Oct 30 01:22:07.138977 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.138858    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="673e2d41-cb67-4956-bb96-959aa37fb924" containerName="istio-proxy"
Oct 30 01:22:07.138977 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.138895    6099 cpu_manager.go:414] "containerMap: removing container" podUID="673e2d41-cb67-4956-bb96-959aa37fb924" containerName="istio-proxy"
Oct 30 01:22:07.138977 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.138904    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="673e2d41-cb67-4956-bb96-959aa37fb924" containerName="istio-proxy"
Oct 30 01:22:07.138977 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.138931    6099 cpu_manager.go:414] "containerMap: removing container" podUID="9af17237-0f9e-435c-b453-ced511ca7425" containerName="revoke-keys"
Oct 30 01:22:07.138977 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.138940    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af17237-0f9e-435c-b453-ced511ca7425" containerName="revoke-keys"
Oct 30 01:22:07.139352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.139137    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="673e2d41-cb67-4956-bb96-959aa37fb924" containerName="istio-proxy"
Oct 30 01:22:07.139352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.139157    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="673e2d41-cb67-4956-bb96-959aa37fb924" containerName="istio-proxy"
Oct 30 01:22:07.139352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.139165    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="87d1ee54-2797-4936-a3a3-44624f7c15c8" containerName="istio-proxy"
Oct 30 01:22:07.139352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.139180    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af17237-0f9e-435c-b453-ced511ca7425" containerName="revoke-keys"
Oct 30 01:22:07.142674 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.142651    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq"
Oct 30 01:22:07.151866 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.151822    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq"]
Oct 30 01:22:07.157363 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.157339    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:22:07.157607 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.157470    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:22:07.311882 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.311842    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305f33b2-e183-44b6-9f89-e90047f939e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq\" (UID: \"305f33b2-e183-44b6-9f89-e90047f939e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq"
Oct 30 01:22:07.312408 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.312086    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305f33b2-e183-44b6-9f89-e90047f939e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq\" (UID: \"305f33b2-e183-44b6-9f89-e90047f939e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq"
Oct 30 01:22:07.312408 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.312143    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mc7n\" (UniqueName: \"kubernetes.io/projected/305f33b2-e183-44b6-9f89-e90047f939e8-kube-api-access-6mc7n\") pod \"maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq\" (UID: \"305f33b2-e183-44b6-9f89-e90047f939e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq"
Oct 30 01:22:07.414218 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.414125    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305f33b2-e183-44b6-9f89-e90047f939e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq\" (UID: \"305f33b2-e183-44b6-9f89-e90047f939e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq"
Oct 30 01:22:07.414218 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.414178    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mc7n\" (UniqueName: \"kubernetes.io/projected/305f33b2-e183-44b6-9f89-e90047f939e8-kube-api-access-6mc7n\") pod \"maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq\" (UID: \"305f33b2-e183-44b6-9f89-e90047f939e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq"
Oct 30 01:22:07.414539 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.414479    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305f33b2-e183-44b6-9f89-e90047f939e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq\" (UID: \"305f33b2-e183-44b6-9f89-e90047f939e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq"
Oct 30 01:22:07.414939 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.414914    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305f33b2-e183-44b6-9f89-e90047f939e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq\" (UID: \"305f33b2-e183-44b6-9f89-e90047f939e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq"
Oct 30 01:22:07.417057 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.417029    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305f33b2-e183-44b6-9f89-e90047f939e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq\" (UID: \"305f33b2-e183-44b6-9f89-e90047f939e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq"
Oct 30 01:22:07.439829 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.439800    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mc7n\" (UniqueName: \"kubernetes.io/projected/305f33b2-e183-44b6-9f89-e90047f939e8-kube-api-access-6mc7n\") pod \"maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq\" (UID: \"305f33b2-e183-44b6-9f89-e90047f939e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq"
Oct 30 01:22:07.479834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.479808    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq"
Oct 30 01:22:07.792809 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:22:07.792776    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305f33b2_e183_44b6_9f89_e90047f939e8.slice/crio-df9a445059ff14ae9f0190903d8a386a54a826ee9522678c3e3219ffc125437e WatchSource:0}: Error finding container df9a445059ff14ae9f0190903d8a386a54a826ee9522678c3e3219ffc125437e: Status 404 returned error can't find the container with id df9a445059ff14ae9f0190903d8a386a54a826ee9522678c3e3219ffc125437e
Oct 30 01:22:07.793374 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:07.793341    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq"]
Oct 30 01:22:08.657150 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:08.657113    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq" event={"ID":"305f33b2-e183-44b6-9f89-e90047f939e8","Type":"ContainerStarted","Data":"d6dd90bdf3c8589d5b960c76a1446c688ffc896022e98adfbf4936b878d8a590"}
Oct 30 01:22:08.657545 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:08.657178    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq" event={"ID":"305f33b2-e183-44b6-9f89-e90047f939e8","Type":"ContainerStarted","Data":"df9a445059ff14ae9f0190903d8a386a54a826ee9522678c3e3219ffc125437e"}
Oct 30 01:22:08.701035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:08.700984    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq" podStartSLOduration=1.700971536 podStartE2EDuration="1.700971536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:08.698074642 +0000 UTC m=+2822.554982396" watchObservedRunningTime="2026-10-30 01:22:08.700971536 +0000 UTC m=+2822.557879290"
Oct 30 01:22:09.665293 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:09.665260    6099 generic.go:356] "Generic (PLEG): container finished" podID="305f33b2-e183-44b6-9f89-e90047f939e8" containerID="d6dd90bdf3c8589d5b960c76a1446c688ffc896022e98adfbf4936b878d8a590" exitCode=0
Oct 30 01:22:09.665699 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:09.665340    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq" event={"ID":"305f33b2-e183-44b6-9f89-e90047f939e8","Type":"ContainerDied","Data":"d6dd90bdf3c8589d5b960c76a1446c688ffc896022e98adfbf4936b878d8a590"}
Oct 30 01:22:10.925778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:10.925222    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq"
Oct 30 01:22:10.976513 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:10.975170    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305f33b2-e183-44b6-9f89-e90047f939e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305f33b2-e183-44b6-9f89-e90047f939e8-maas-api-openshift-ca\") pod \"305f33b2-e183-44b6-9f89-e90047f939e8\" (UID: \"305f33b2-e183-44b6-9f89-e90047f939e8\") "
Oct 30 01:22:10.976513 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:10.975314    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305f33b2-e183-44b6-9f89-e90047f939e8-kube-api-access-6mc7n\" (UniqueName: \"kubernetes.io/projected/305f33b2-e183-44b6-9f89-e90047f939e8-kube-api-access-6mc7n\") pod \"305f33b2-e183-44b6-9f89-e90047f939e8\" (UID: \"305f33b2-e183-44b6-9f89-e90047f939e8\") "
Oct 30 01:22:10.976513 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:10.975757    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305f33b2-e183-44b6-9f89-e90047f939e8-maas-api-openshift-ca" pod "305f33b2-e183-44b6-9f89-e90047f939e8" (UID: "305f33b2-e183-44b6-9f89-e90047f939e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:10.976513 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:10.975836    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/305f33b2-e183-44b6-9f89-e90047f939e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305f33b2-e183-44b6-9f89-e90047f939e8-maas-api-xks-ca\") pod \"305f33b2-e183-44b6-9f89-e90047f939e8\" (UID: \"305f33b2-e183-44b6-9f89-e90047f939e8\") "
Oct 30 01:22:10.976920 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:10.976578    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305f33b2-e183-44b6-9f89-e90047f939e8-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:10.980199 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:10.978829    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/305f33b2-e183-44b6-9f89-e90047f939e8-maas-api-xks-ca" pod "305f33b2-e183-44b6-9f89-e90047f939e8" (UID: "305f33b2-e183-44b6-9f89-e90047f939e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:10.983108 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:10.982921    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305f33b2-e183-44b6-9f89-e90047f939e8-kube-api-access-6mc7n" pod "305f33b2-e183-44b6-9f89-e90047f939e8" (UID: "305f33b2-e183-44b6-9f89-e90047f939e8"). InnerVolumeSpecName "kube-api-access-6mc7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:11.078566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.078530    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mc7n\" (UniqueName: \"kubernetes.io/projected/305f33b2-e183-44b6-9f89-e90047f939e8-kube-api-access-6mc7n\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:11.078566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.078563    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305f33b2-e183-44b6-9f89-e90047f939e8-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:11.688050 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.688015    6099 generic.go:356] "Generic (PLEG): container finished" podID="bdf12444-9b7c-4721-900b-44b11df44dd1" containerID="cc1b96ec9b18656aeac10b69e06acc1438e7595307bb741bb3a5ef4ab8072137" exitCode=0
Oct 30 01:22:11.688285 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.688261    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72" event={"ID":"bdf12444-9b7c-4721-900b-44b11df44dd1","Type":"ContainerDied","Data":"cc1b96ec9b18656aeac10b69e06acc1438e7595307bb741bb3a5ef4ab8072137"}
Oct 30 01:22:11.688402 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.688332    6099 scope.go:122] "RemoveContainer" containerID="99574e2a6108acfee0ee05fcd0952a3b7d812f8531436b457ee9397de3e2a065"
Oct 30 01:22:11.690429 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.690293    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq" event={"ID":"305f33b2-e183-44b6-9f89-e90047f939e8","Type":"ContainerDied","Data":"df9a445059ff14ae9f0190903d8a386a54a826ee9522678c3e3219ffc125437e"}
Oct 30 01:22:11.690429 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.690345    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df9a445059ff14ae9f0190903d8a386a54a826ee9522678c3e3219ffc125437e"
Oct 30 01:22:11.690429 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.690410    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq"
Oct 30 01:22:11.700385 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.700367    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 01:22:11.787842 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.787810    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-workload-certs\") pod \"bdf12444-9b7c-4721-900b-44b11df44dd1\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") "
Oct 30 01:22:11.788037 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.787869    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-token\" (UniqueName: \"kubernetes.io/projected/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-token\") pod \"bdf12444-9b7c-4721-900b-44b11df44dd1\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") "
Oct 30 01:22:11.788037 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.787980    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-credential-socket\") pod \"bdf12444-9b7c-4721-900b-44b11df44dd1\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") "
Oct 30 01:22:11.788037 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.788016    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdf12444-9b7c-4721-900b-44b11df44dd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf12444-9b7c-4721-900b-44b11df44dd1-istiod-ca-cert\") pod \"bdf12444-9b7c-4721-900b-44b11df44dd1\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") "
Oct 30 01:22:11.788221 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.788103    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-envoy\") pod \"bdf12444-9b7c-4721-900b-44b11df44dd1\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") "
Oct 30 01:22:11.788221 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.788114    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-workload-certs" pod "bdf12444-9b7c-4721-900b-44b11df44dd1" (UID: "bdf12444-9b7c-4721-900b-44b11df44dd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:11.788331 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.788237    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-data\") pod \"bdf12444-9b7c-4721-900b-44b11df44dd1\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") "
Oct 30 01:22:11.788331 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.788274    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-podinfo\") pod \"bdf12444-9b7c-4721-900b-44b11df44dd1\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") "
Oct 30 01:22:11.788502 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.788392    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-credential-socket" pod "bdf12444-9b7c-4721-900b-44b11df44dd1" (UID: "bdf12444-9b7c-4721-900b-44b11df44dd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:11.788502 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.788401    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf12444-9b7c-4721-900b-44b11df44dd1-kube-api-access-7gjh7\" (UniqueName: \"kubernetes.io/projected/bdf12444-9b7c-4721-900b-44b11df44dd1-kube-api-access-7gjh7\") pod \"bdf12444-9b7c-4721-900b-44b11df44dd1\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") "
Oct 30 01:22:11.788502 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.788496    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-workload-socket\") pod \"bdf12444-9b7c-4721-900b-44b11df44dd1\" (UID: \"bdf12444-9b7c-4721-900b-44b11df44dd1\") "
Oct 30 01:22:11.788682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.788574    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-data" pod "bdf12444-9b7c-4721-900b-44b11df44dd1" (UID: "bdf12444-9b7c-4721-900b-44b11df44dd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:11.788682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.788496    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf12444-9b7c-4721-900b-44b11df44dd1-istiod-ca-cert" pod "bdf12444-9b7c-4721-900b-44b11df44dd1" (UID: "bdf12444-9b7c-4721-900b-44b11df44dd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:11.788904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.788882    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-workload-socket" pod "bdf12444-9b7c-4721-900b-44b11df44dd1" (UID: "bdf12444-9b7c-4721-900b-44b11df44dd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:11.789289 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.789259    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:11.789411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.789291    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:11.789411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.789306    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:11.789411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.789320    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:11.789411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.789335    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf12444-9b7c-4721-900b-44b11df44dd1-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:11.790606 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.790582    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-envoy" pod "bdf12444-9b7c-4721-900b-44b11df44dd1" (UID: "bdf12444-9b7c-4721-900b-44b11df44dd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:11.790942 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.790924    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-token" pod "bdf12444-9b7c-4721-900b-44b11df44dd1" (UID: "bdf12444-9b7c-4721-900b-44b11df44dd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:11.791345 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.791326    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf12444-9b7c-4721-900b-44b11df44dd1-kube-api-access-7gjh7" pod "bdf12444-9b7c-4721-900b-44b11df44dd1" (UID: "bdf12444-9b7c-4721-900b-44b11df44dd1"). InnerVolumeSpecName "kube-api-access-7gjh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:11.791598 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.791575    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-podinfo" pod "bdf12444-9b7c-4721-900b-44b11df44dd1" (UID: "bdf12444-9b7c-4721-900b-44b11df44dd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:22:11.890987 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.890954    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:11.890987 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.890985    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gjh7\" (UniqueName: \"kubernetes.io/projected/bdf12444-9b7c-4721-900b-44b11df44dd1-kube-api-access-7gjh7\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:11.890987 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.890995    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:11.891229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.891004    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf12444-9b7c-4721-900b-44b11df44dd1-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:11.891660 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:11.891637    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"]
Oct 30 01:22:12.657729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.657677    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq"]
Oct 30 01:22:12.659315 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.659286    6099 cpu_manager.go:414] "containerMap: removing container" podUID="bdf12444-9b7c-4721-900b-44b11df44dd1" containerName="istio-proxy"
Oct 30 01:22:12.659483 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.659470    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf12444-9b7c-4721-900b-44b11df44dd1" containerName="istio-proxy"
Oct 30 01:22:12.659595 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.659586    6099 cpu_manager.go:414] "containerMap: removing container" podUID="305f33b2-e183-44b6-9f89-e90047f939e8" containerName="revoke-keys"
Oct 30 01:22:12.659668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.659659    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305f33b2-e183-44b6-9f89-e90047f939e8" containerName="revoke-keys"
Oct 30 01:22:12.676782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.660004    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="305f33b2-e183-44b6-9f89-e90047f939e8" containerName="revoke-keys"
Oct 30 01:22:12.678449 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.678420    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf12444-9b7c-4721-900b-44b11df44dd1" containerName="istio-proxy"
Oct 30 01:22:12.678631 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.678618    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf12444-9b7c-4721-900b-44b11df44dd1" containerName="istio-proxy"
Oct 30 01:22:12.683175 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.683152    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq"
Oct 30 01:22:12.715295 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.715261    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq"]
Oct 30 01:22:12.715775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.715742    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq\" (UID: \"e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq"
Oct 30 01:22:12.716041 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.715823    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbxlg\" (UniqueName: \"kubernetes.io/projected/e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55-kube-api-access-zbxlg\") pod \"payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq\" (UID: \"e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq"
Oct 30 01:22:12.719088 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.718946    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3e9921\""
Oct 30 01:22:12.719648 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.719627    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3e9921-dockercfg-f782b\""
Oct 30 01:22:12.723326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.721926    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72" event={"ID":"bdf12444-9b7c-4721-900b-44b11df44dd1","Type":"ContainerDied","Data":"f39897083432f617c459f65b49fd36917534fc53cc40cdcfea89a8a2e24596d9"}
Oct 30 01:22:12.723326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.721984    6099 scope.go:122] "RemoveContainer" containerID="cc1b96ec9b18656aeac10b69e06acc1438e7595307bb741bb3a5ef4ab8072137"
Oct 30 01:22:12.723326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.722089    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"
Oct 30 01:22:12.818585 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.818545    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbxlg\" (UniqueName: \"kubernetes.io/projected/e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55-kube-api-access-zbxlg\") pod \"payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq\" (UID: \"e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq"
Oct 30 01:22:12.818899 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.818882    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq\" (UID: \"e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq"
Oct 30 01:22:12.819534 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.819514    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq\" (UID: \"e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq"
Oct 30 01:22:12.826252 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.826218    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"]
Oct 30 01:22:12.830822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.830796    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62f5d1-openshift-default-69fc57b585-nlc72"]
Oct 30 01:22:12.842472 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:12.842444    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbxlg\" (UniqueName: \"kubernetes.io/projected/e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55-kube-api-access-zbxlg\") pod \"payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq\" (UID: \"e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq"
Oct 30 01:22:13.068916 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:13.068882    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq"
Oct 30 01:22:13.374610 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:13.374581    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq"]
Oct 30 01:22:13.375339 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:22:13.375310    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ff88a6_ef5c_44d7_ab3c_93d3d2e8cd55.slice/crio-b4a66a625cb6a001c987e8f558b0def463da8c1bc74d543c7c936daf03b217f1 WatchSource:0}: Error finding container b4a66a625cb6a001c987e8f558b0def463da8c1bc74d543c7c936daf03b217f1: Status 404 returned error can't find the container with id b4a66a625cb6a001c987e8f558b0def463da8c1bc74d543c7c936daf03b217f1
Oct 30 01:22:13.732337 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:13.732234    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq" event={"ID":"e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55","Type":"ContainerStarted","Data":"01e4eac101cbaddbfb9bf52ab5a1ed2ecbae6d080e7101922f54a7aaf06c27b3"}
Oct 30 01:22:13.732337 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:13.732296    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq" event={"ID":"e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55","Type":"ContainerStarted","Data":"b4a66a625cb6a001c987e8f558b0def463da8c1bc74d543c7c936daf03b217f1"}
Oct 30 01:22:13.732337 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:13.732324    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq"
Oct 30 01:22:13.784726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:13.784661    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq" podStartSLOduration=1.784644798 podStartE2EDuration="1.784644798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:13.782642133 +0000 UTC m=+2827.639549889" watchObservedRunningTime="2026-10-30 01:22:13.784644798 +0000 UTC m=+2827.641552555"
Oct 30 01:22:14.262992 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:14.262959    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdf12444-9b7c-4721-900b-44b11df44dd1" path="/var/lib/kubelet/pods/bdf12444-9b7c-4721-900b-44b11df44dd1/volumes"
Oct 30 01:22:16.651662 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:22:16.651628    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:22:17.520660 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:17.520624    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt"]
Oct 30 01:22:17.553955 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:17.553927    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr"]
Oct 30 01:22:19.322525 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.322479    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" containerID="cri-o://09714262f7900443f6aaa3d428969b28fc73e84e7522cd292e1ecab94dd25078" gracePeriod=45
Oct 30 01:22:19.347253 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.347226    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": dial tcp 10.134.0.68:8000: connect: connection refused"
Oct 30 01:22:19.606959 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.606933    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 01:22:19.721076 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.721044    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-dshm\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-dshm\") pod \"69232e0b-1728-4658-874f-e79bbee81329\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") "
Oct 30 01:22:19.721227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.721083    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-home\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-home\") pod \"69232e0b-1728-4658-874f-e79bbee81329\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") "
Oct 30 01:22:19.721290 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.721228    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69232e0b-1728-4658-874f-e79bbee81329-tls-certs\" (UniqueName: \"kubernetes.io/secret/69232e0b-1728-4658-874f-e79bbee81329-tls-certs\") pod \"69232e0b-1728-4658-874f-e79bbee81329\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") "
Oct 30 01:22:19.721350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.721332    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-tmp-dir\") pod \"69232e0b-1728-4658-874f-e79bbee81329\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") "
Oct 30 01:22:19.721393 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.721366    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-home" pod "69232e0b-1728-4658-874f-e79bbee81329" (UID: "69232e0b-1728-4658-874f-e79bbee81329"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:19.721489 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.721470    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69232e0b-1728-4658-874f-e79bbee81329-kube-api-access-8cbwh\" (UniqueName: \"kubernetes.io/projected/69232e0b-1728-4658-874f-e79bbee81329-kube-api-access-8cbwh\") pod \"69232e0b-1728-4658-874f-e79bbee81329\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") "
Oct 30 01:22:19.721574 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.721522    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-model-cache\") pod \"69232e0b-1728-4658-874f-e79bbee81329\" (UID: \"69232e0b-1728-4658-874f-e79bbee81329\") "
Oct 30 01:22:19.721632 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.721589    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-tmp-dir" pod "69232e0b-1728-4658-874f-e79bbee81329" (UID: "69232e0b-1728-4658-874f-e79bbee81329"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:19.721814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.721783    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-model-cache" pod "69232e0b-1728-4658-874f-e79bbee81329" (UID: "69232e0b-1728-4658-874f-e79bbee81329"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:19.722350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.722328    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:19.722413 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.722357    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:19.722413 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.722371    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:19.723308 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.723284    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-dshm" pod "69232e0b-1728-4658-874f-e79bbee81329" (UID: "69232e0b-1728-4658-874f-e79bbee81329"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:19.723450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.723424    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69232e0b-1728-4658-874f-e79bbee81329-tls-certs" pod "69232e0b-1728-4658-874f-e79bbee81329" (UID: "69232e0b-1728-4658-874f-e79bbee81329"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:19.723543 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.723525    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69232e0b-1728-4658-874f-e79bbee81329-kube-api-access-8cbwh" pod "69232e0b-1728-4658-874f-e79bbee81329" (UID: "69232e0b-1728-4658-874f-e79bbee81329"). InnerVolumeSpecName "kube-api-access-8cbwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:19.797001 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.796862    6099 generic.go:356] "Generic (PLEG): container finished" podID="69232e0b-1728-4658-874f-e79bbee81329" containerID="09714262f7900443f6aaa3d428969b28fc73e84e7522cd292e1ecab94dd25078" exitCode=0
Oct 30 01:22:19.797001 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.796904    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" event={"ID":"69232e0b-1728-4658-874f-e79bbee81329","Type":"ContainerDied","Data":"09714262f7900443f6aaa3d428969b28fc73e84e7522cd292e1ecab94dd25078"}
Oct 30 01:22:19.797001 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.796925    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"
Oct 30 01:22:19.797001 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.796944    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq" event={"ID":"69232e0b-1728-4658-874f-e79bbee81329","Type":"ContainerDied","Data":"ac1e223c810b85353d22118239f9c460a2775a805418c5ef76d26cea49f1e874"}
Oct 30 01:22:19.797001 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.796964    6099 scope.go:122] "RemoveContainer" containerID="09714262f7900443f6aaa3d428969b28fc73e84e7522cd292e1ecab94dd25078"
Oct 30 01:22:19.813591 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.813568    6099 scope.go:122] "RemoveContainer" containerID="09714262f7900443f6aaa3d428969b28fc73e84e7522cd292e1ecab94dd25078"
Oct 30 01:22:19.814031 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:22:19.814009    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09714262f7900443f6aaa3d428969b28fc73e84e7522cd292e1ecab94dd25078\": container with ID starting with 09714262f7900443f6aaa3d428969b28fc73e84e7522cd292e1ecab94dd25078 not found: ID does not exist" containerID="09714262f7900443f6aaa3d428969b28fc73e84e7522cd292e1ecab94dd25078"
Oct 30 01:22:19.814135 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.814044    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09714262f7900443f6aaa3d428969b28fc73e84e7522cd292e1ecab94dd25078"} err="failed to get container status \"09714262f7900443f6aaa3d428969b28fc73e84e7522cd292e1ecab94dd25078\": rpc error: code = NotFound desc = could not find container \"09714262f7900443f6aaa3d428969b28fc73e84e7522cd292e1ecab94dd25078\": container with ID starting with 09714262f7900443f6aaa3d428969b28fc73e84e7522cd292e1ecab94dd25078 not found: ID does not exist"
Oct 30 01:22:19.824478 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.824450    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69232e0b-1728-4658-874f-e79bbee81329-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:19.824569 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.824484    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69232e0b-1728-4658-874f-e79bbee81329-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:19.824569 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.824500    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cbwh\" (UniqueName: \"kubernetes.io/projected/69232e0b-1728-4658-874f-e79bbee81329-kube-api-access-8cbwh\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:19.855104 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.855037    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"]
Oct 30 01:22:19.858581 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:19.858557    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6add35/distinct-w4-6add35-kserve-7c57f95748-gplqq"]
Oct 30 01:22:20.262133 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:20.262061    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69232e0b-1728-4658-874f-e79bbee81329" path="/var/lib/kubelet/pods/69232e0b-1728-4658-874f-e79bbee81329/volumes"
Oct 30 01:22:22.521380 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:22.521339    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt" podUID="61780fe7-50f0-4f30-989c-03e165d46adf" containerName="payload-processing" containerID="cri-o://e813d54d5bf107aa6a67f4c8b4179365ea39b8456237216e49c85027e710b8bb" gracePeriod=25
Oct 30 01:22:22.554296 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:22.554258    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr" podUID="354c452d-1616-4d33-9d18-a44f6fc76366" containerName="payload-pre-processing" containerID="cri-o://030ee7d7bc1dfd3d6150d889b785b249161e12d8010c6903639a51550b636745" gracePeriod=25
Oct 30 01:22:24.744009 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:24.743984    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq"
Oct 30 01:22:26.913637 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:26.913599    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main" containerID="cri-o://438d0e365769ef86eb391a1c4a9580cf527413e309ff1f5d9238af1eeb9ff251" gracePeriod=45
Oct 30 01:22:27.185219 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.185158    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 01:22:27.352537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.352504    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-tmp-dir\") pod \"ef896512-2f16-4fd1-a732-cf9c232994e0\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") "
Oct 30 01:22:27.352763 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.352716    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-home\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-home\") pod \"ef896512-2f16-4fd1-a732-cf9c232994e0\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") "
Oct 30 01:22:27.352852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.352807    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef896512-2f16-4fd1-a732-cf9c232994e0-kube-api-access-vxfw7\" (UniqueName: \"kubernetes.io/projected/ef896512-2f16-4fd1-a732-cf9c232994e0-kube-api-access-vxfw7\") pod \"ef896512-2f16-4fd1-a732-cf9c232994e0\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") "
Oct 30 01:22:27.352913 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.352864    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-model-cache\") pod \"ef896512-2f16-4fd1-a732-cf9c232994e0\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") "
Oct 30 01:22:27.352913 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.352870    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-tmp-dir" pod "ef896512-2f16-4fd1-a732-cf9c232994e0" (UID: "ef896512-2f16-4fd1-a732-cf9c232994e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:27.353036 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.352954    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-dshm\") pod \"ef896512-2f16-4fd1-a732-cf9c232994e0\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") "
Oct 30 01:22:27.353036 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.352985    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef896512-2f16-4fd1-a732-cf9c232994e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef896512-2f16-4fd1-a732-cf9c232994e0-tls-certs\") pod \"ef896512-2f16-4fd1-a732-cf9c232994e0\" (UID: \"ef896512-2f16-4fd1-a732-cf9c232994e0\") "
Oct 30 01:22:27.353154 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.353121    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-model-cache" pod "ef896512-2f16-4fd1-a732-cf9c232994e0" (UID: "ef896512-2f16-4fd1-a732-cf9c232994e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:27.353743 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.353697    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:27.353743 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.353744    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:27.354759 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.354735    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-home" pod "ef896512-2f16-4fd1-a732-cf9c232994e0" (UID: "ef896512-2f16-4fd1-a732-cf9c232994e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:27.356460 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.356430    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-dshm" pod "ef896512-2f16-4fd1-a732-cf9c232994e0" (UID: "ef896512-2f16-4fd1-a732-cf9c232994e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:27.369157 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.369114    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef896512-2f16-4fd1-a732-cf9c232994e0-kube-api-access-vxfw7" pod "ef896512-2f16-4fd1-a732-cf9c232994e0" (UID: "ef896512-2f16-4fd1-a732-cf9c232994e0"). InnerVolumeSpecName "kube-api-access-vxfw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:27.369157 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.369123    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef896512-2f16-4fd1-a732-cf9c232994e0-tls-certs" pod "ef896512-2f16-4fd1-a732-cf9c232994e0" (UID: "ef896512-2f16-4fd1-a732-cf9c232994e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:27.455576 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.455493    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:27.455576 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.455521    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxfw7\" (UniqueName: \"kubernetes.io/projected/ef896512-2f16-4fd1-a732-cf9c232994e0-kube-api-access-vxfw7\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:27.455576 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.455535    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef896512-2f16-4fd1-a732-cf9c232994e0-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:27.455576 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.455546    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef896512-2f16-4fd1-a732-cf9c232994e0-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:27.864245 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.864212    6099 generic.go:356] "Generic (PLEG): container finished" podID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerID="438d0e365769ef86eb391a1c4a9580cf527413e309ff1f5d9238af1eeb9ff251" exitCode=0
Oct 30 01:22:27.864439 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.864266    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"
Oct 30 01:22:27.864439 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.864289    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" event={"ID":"ef896512-2f16-4fd1-a732-cf9c232994e0","Type":"ContainerDied","Data":"438d0e365769ef86eb391a1c4a9580cf527413e309ff1f5d9238af1eeb9ff251"}
Oct 30 01:22:27.864439 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.864336    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw" event={"ID":"ef896512-2f16-4fd1-a732-cf9c232994e0","Type":"ContainerDied","Data":"ae0de0e10fced29a01287b6a240aaff5e0e13b11f200e1d4c75e12ef95e43d80"}
Oct 30 01:22:27.864439 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.864354    6099 scope.go:122] "RemoveContainer" containerID="438d0e365769ef86eb391a1c4a9580cf527413e309ff1f5d9238af1eeb9ff251"
Oct 30 01:22:27.875566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.875550    6099 scope.go:122] "RemoveContainer" containerID="438d0e365769ef86eb391a1c4a9580cf527413e309ff1f5d9238af1eeb9ff251"
Oct 30 01:22:27.875857 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:22:27.875834    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"438d0e365769ef86eb391a1c4a9580cf527413e309ff1f5d9238af1eeb9ff251\": container with ID starting with 438d0e365769ef86eb391a1c4a9580cf527413e309ff1f5d9238af1eeb9ff251 not found: ID does not exist" containerID="438d0e365769ef86eb391a1c4a9580cf527413e309ff1f5d9238af1eeb9ff251"
Oct 30 01:22:27.875942 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.875865    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"438d0e365769ef86eb391a1c4a9580cf527413e309ff1f5d9238af1eeb9ff251"} err="failed to get container status \"438d0e365769ef86eb391a1c4a9580cf527413e309ff1f5d9238af1eeb9ff251\": rpc error: code = NotFound desc = could not find container \"438d0e365769ef86eb391a1c4a9580cf527413e309ff1f5d9238af1eeb9ff251\": container with ID starting with 438d0e365769ef86eb391a1c4a9580cf527413e309ff1f5d9238af1eeb9ff251 not found: ID does not exist"
Oct 30 01:22:27.917589 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.917564    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"]
Oct 30 01:22:27.928953 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:27.928929    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2b4105/premium-w0-2b4105-kserve-6f6b659d9d-99vzw"]
Oct 30 01:22:28.262466 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:28.262389    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" path="/var/lib/kubelet/pods/ef896512-2f16-4fd1-a732-cf9c232994e0/volumes"
Oct 30 01:22:29.442274 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:29.442234    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt" podUID="61780fe7-50f0-4f30-989c-03e165d46adf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:22:29.442690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:29.442238    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr" podUID="354c452d-1616-4d33-9d18-a44f6fc76366" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:22:31.113738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.113673    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj"]
Oct 30 01:22:31.114399 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.114381    6099 cpu_manager.go:414] "containerMap: removing container" podUID="bdf12444-9b7c-4721-900b-44b11df44dd1" containerName="istio-proxy"
Oct 30 01:22:31.114399 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.114399    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf12444-9b7c-4721-900b-44b11df44dd1" containerName="istio-proxy"
Oct 30 01:22:31.114535 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.114427    6099 cpu_manager.go:414] "containerMap: removing container" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main"
Oct 30 01:22:31.114535 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.114433    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main"
Oct 30 01:22:31.114535 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.114444    6099 cpu_manager.go:414] "containerMap: removing container" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main"
Oct 30 01:22:31.114535 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.114449    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main"
Oct 30 01:22:31.114535 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.114526    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef896512-2f16-4fd1-a732-cf9c232994e0" containerName="main"
Oct 30 01:22:31.114826 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.114545    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="69232e0b-1728-4658-874f-e79bbee81329" containerName="main"
Oct 30 01:22:31.119618 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.119597    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj"
Oct 30 01:22:31.147699 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.147629    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:22:31.147945 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.147639    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:22:31.154884 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.154855    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj"]
Oct 30 01:22:31.202050 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.202009    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a0c9803-5wpzj\" (UID: \"8f79a646-0479-486f-8b0b-dda8e7ed0ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj"
Oct 30 01:22:31.202235 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.202134    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a0c9803-5wpzj\" (UID: \"8f79a646-0479-486f-8b0b-dda8e7ed0ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj"
Oct 30 01:22:31.202433 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.202412    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lpqv\" (UniqueName: \"kubernetes.io/projected/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-kube-api-access-8lpqv\") pod \"maas-api-revoke-sub-3a0c9803-5wpzj\" (UID: \"8f79a646-0479-486f-8b0b-dda8e7ed0ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj"
Oct 30 01:22:31.303936 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.303899    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a0c9803-5wpzj\" (UID: \"8f79a646-0479-486f-8b0b-dda8e7ed0ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj"
Oct 30 01:22:31.304146 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.303950    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a0c9803-5wpzj\" (UID: \"8f79a646-0479-486f-8b0b-dda8e7ed0ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj"
Oct 30 01:22:31.304223 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.304175    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lpqv\" (UniqueName: \"kubernetes.io/projected/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-kube-api-access-8lpqv\") pod \"maas-api-revoke-sub-3a0c9803-5wpzj\" (UID: \"8f79a646-0479-486f-8b0b-dda8e7ed0ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj"
Oct 30 01:22:31.304722 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.304686    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a0c9803-5wpzj\" (UID: \"8f79a646-0479-486f-8b0b-dda8e7ed0ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj"
Oct 30 01:22:31.306307 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.306286    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a0c9803-5wpzj\" (UID: \"8f79a646-0479-486f-8b0b-dda8e7ed0ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj"
Oct 30 01:22:31.327353 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.327319    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lpqv\" (UniqueName: \"kubernetes.io/projected/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-kube-api-access-8lpqv\") pod \"maas-api-revoke-sub-3a0c9803-5wpzj\" (UID: \"8f79a646-0479-486f-8b0b-dda8e7ed0ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj"
Oct 30 01:22:31.444152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.444082    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj"
Oct 30 01:22:31.748530 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.748500    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj"]
Oct 30 01:22:31.748656 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:22:31.748576    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f79a646_0479_486f_8b0b_dda8e7ed0ac4.slice/crio-52d429f7dafd800519a1f110e4ec4aadb106da1e6251fc7e021778f5e08ba9f3 WatchSource:0}: Error finding container 52d429f7dafd800519a1f110e4ec4aadb106da1e6251fc7e021778f5e08ba9f3: Status 404 returned error can't find the container with id 52d429f7dafd800519a1f110e4ec4aadb106da1e6251fc7e021778f5e08ba9f3
Oct 30 01:22:31.894752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:31.894688    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj" event={"ID":"8f79a646-0479-486f-8b0b-dda8e7ed0ac4","Type":"ContainerStarted","Data":"52d429f7dafd800519a1f110e4ec4aadb106da1e6251fc7e021778f5e08ba9f3"}
Oct 30 01:22:32.903516 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:32.903479    6099 generic.go:356] "Generic (PLEG): container finished" podID="8f79a646-0479-486f-8b0b-dda8e7ed0ac4" containerID="8143995498b045c1d9897d7ec39c81537ec2e0bcb1a1f2fb10c014d6162ffe07" exitCode=0
Oct 30 01:22:32.903979 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:32.903563    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj" event={"ID":"8f79a646-0479-486f-8b0b-dda8e7ed0ac4","Type":"ContainerDied","Data":"8143995498b045c1d9897d7ec39c81537ec2e0bcb1a1f2fb10c014d6162ffe07"}
Oct 30 01:22:34.072858 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:34.072832    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj"
Oct 30 01:22:34.147352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:34.147319    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-maas-api-xks-ca\") pod \"8f79a646-0479-486f-8b0b-dda8e7ed0ac4\" (UID: \"8f79a646-0479-486f-8b0b-dda8e7ed0ac4\") "
Oct 30 01:22:34.147525 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:34.147488    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-kube-api-access-8lpqv\" (UniqueName: \"kubernetes.io/projected/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-kube-api-access-8lpqv\") pod \"8f79a646-0479-486f-8b0b-dda8e7ed0ac4\" (UID: \"8f79a646-0479-486f-8b0b-dda8e7ed0ac4\") "
Oct 30 01:22:34.147640 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:34.147621    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-maas-api-openshift-ca\") pod \"8f79a646-0479-486f-8b0b-dda8e7ed0ac4\" (UID: \"8f79a646-0479-486f-8b0b-dda8e7ed0ac4\") "
Oct 30 01:22:34.148063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:34.148034    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-maas-api-openshift-ca" pod "8f79a646-0479-486f-8b0b-dda8e7ed0ac4" (UID: "8f79a646-0479-486f-8b0b-dda8e7ed0ac4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:34.148358 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:34.148335    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:34.149786 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:34.149758    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-maas-api-xks-ca" pod "8f79a646-0479-486f-8b0b-dda8e7ed0ac4" (UID: "8f79a646-0479-486f-8b0b-dda8e7ed0ac4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:34.150152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:34.150129    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-kube-api-access-8lpqv" pod "8f79a646-0479-486f-8b0b-dda8e7ed0ac4" (UID: "8f79a646-0479-486f-8b0b-dda8e7ed0ac4"). InnerVolumeSpecName "kube-api-access-8lpqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:34.250070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:34.249990    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:34.250070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:34.250020    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lpqv\" (UniqueName: \"kubernetes.io/projected/8f79a646-0479-486f-8b0b-dda8e7ed0ac4-kube-api-access-8lpqv\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:34.918678 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:34.918650    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj"
Oct 30 01:22:34.918872 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:34.918688    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj" event={"ID":"8f79a646-0479-486f-8b0b-dda8e7ed0ac4","Type":"ContainerDied","Data":"52d429f7dafd800519a1f110e4ec4aadb106da1e6251fc7e021778f5e08ba9f3"}
Oct 30 01:22:34.918872 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:34.918771    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52d429f7dafd800519a1f110e4ec4aadb106da1e6251fc7e021778f5e08ba9f3"
Oct 30 01:22:34.991319 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:34.991279    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:22:36.170951 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.170918    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"]
Oct 30 01:22:36.171345 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.171116    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q" podUID="1272be00-9aa6-4cb1-9d1e-74cfe0164569" containerName="maas-api" containerID="cri-o://7a69e9c3ef70c5d906be29da9edd8ceb7846dbdb49dd0da2c50cda004103d597" gracePeriod=30
Oct 30 01:22:36.184369 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.184340    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q" podUID="1272be00-9aa6-4cb1-9d1e-74cfe0164569" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.1.46:8443/health\": dial tcp 10.134.1.46:8443: connect: connection refused"
Oct 30 01:22:36.489500 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.489472    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"
Oct 30 01:22:36.589582 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.589554    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1272be00-9aa6-4cb1-9d1e-74cfe0164569-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1272be00-9aa6-4cb1-9d1e-74cfe0164569-openshift-service-ca\") pod \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\" (UID: \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\") "
Oct 30 01:22:36.589758 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.589675    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1272be00-9aa6-4cb1-9d1e-74cfe0164569-kube-api-access-txlkg\" (UniqueName: \"kubernetes.io/projected/1272be00-9aa6-4cb1-9d1e-74cfe0164569-kube-api-access-txlkg\") pod \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\" (UID: \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\") "
Oct 30 01:22:36.589831 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.589813    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1272be00-9aa6-4cb1-9d1e-74cfe0164569-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1272be00-9aa6-4cb1-9d1e-74cfe0164569-maas-api-tls\") pod \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\" (UID: \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\") "
Oct 30 01:22:36.589886 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.589857    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1272be00-9aa6-4cb1-9d1e-74cfe0164569-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1272be00-9aa6-4cb1-9d1e-74cfe0164569-metrics-certs\") pod \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\" (UID: \"1272be00-9aa6-4cb1-9d1e-74cfe0164569\") "
Oct 30 01:22:36.590010 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.589990    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1272be00-9aa6-4cb1-9d1e-74cfe0164569-openshift-service-ca" pod "1272be00-9aa6-4cb1-9d1e-74cfe0164569" (UID: "1272be00-9aa6-4cb1-9d1e-74cfe0164569"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:36.590445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.590423    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1272be00-9aa6-4cb1-9d1e-74cfe0164569-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:36.591852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.591828    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1272be00-9aa6-4cb1-9d1e-74cfe0164569-kube-api-access-txlkg" pod "1272be00-9aa6-4cb1-9d1e-74cfe0164569" (UID: "1272be00-9aa6-4cb1-9d1e-74cfe0164569"). InnerVolumeSpecName "kube-api-access-txlkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:36.591965 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.591955    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1272be00-9aa6-4cb1-9d1e-74cfe0164569-maas-api-tls" pod "1272be00-9aa6-4cb1-9d1e-74cfe0164569" (UID: "1272be00-9aa6-4cb1-9d1e-74cfe0164569"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:36.592013 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.591975    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1272be00-9aa6-4cb1-9d1e-74cfe0164569-metrics-certs" pod "1272be00-9aa6-4cb1-9d1e-74cfe0164569" (UID: "1272be00-9aa6-4cb1-9d1e-74cfe0164569"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:36.692417 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.692389    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txlkg\" (UniqueName: \"kubernetes.io/projected/1272be00-9aa6-4cb1-9d1e-74cfe0164569-kube-api-access-txlkg\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:36.692417 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.692415    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1272be00-9aa6-4cb1-9d1e-74cfe0164569-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:36.692625 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.692426    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1272be00-9aa6-4cb1-9d1e-74cfe0164569-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:36.934966 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.934926    6099 generic.go:356] "Generic (PLEG): container finished" podID="1272be00-9aa6-4cb1-9d1e-74cfe0164569" containerID="7a69e9c3ef70c5d906be29da9edd8ceb7846dbdb49dd0da2c50cda004103d597" exitCode=0
Oct 30 01:22:36.935143 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.935083    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q" event={"ID":"1272be00-9aa6-4cb1-9d1e-74cfe0164569","Type":"ContainerDied","Data":"7a69e9c3ef70c5d906be29da9edd8ceb7846dbdb49dd0da2c50cda004103d597"}
Oct 30 01:22:36.935143 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.935133    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q" event={"ID":"1272be00-9aa6-4cb1-9d1e-74cfe0164569","Type":"ContainerDied","Data":"1e7be587a83560dade452ff32cee6d14871fa6f140af1da8798f29ecbc6603c6"}
Oct 30 01:22:36.935289 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.935162    6099 scope.go:122] "RemoveContainer" containerID="7a69e9c3ef70c5d906be29da9edd8ceb7846dbdb49dd0da2c50cda004103d597"
Oct 30 01:22:36.935374 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.935350    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"
Oct 30 01:22:36.953862 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.953820    6099 scope.go:122] "RemoveContainer" containerID="7a69e9c3ef70c5d906be29da9edd8ceb7846dbdb49dd0da2c50cda004103d597"
Oct 30 01:22:36.954143 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:22:36.954117    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a69e9c3ef70c5d906be29da9edd8ceb7846dbdb49dd0da2c50cda004103d597\": container with ID starting with 7a69e9c3ef70c5d906be29da9edd8ceb7846dbdb49dd0da2c50cda004103d597 not found: ID does not exist" containerID="7a69e9c3ef70c5d906be29da9edd8ceb7846dbdb49dd0da2c50cda004103d597"
Oct 30 01:22:36.954207 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.954153    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a69e9c3ef70c5d906be29da9edd8ceb7846dbdb49dd0da2c50cda004103d597"} err="failed to get container status \"7a69e9c3ef70c5d906be29da9edd8ceb7846dbdb49dd0da2c50cda004103d597\": rpc error: code = NotFound desc = could not find container \"7a69e9c3ef70c5d906be29da9edd8ceb7846dbdb49dd0da2c50cda004103d597\": container with ID starting with 7a69e9c3ef70c5d906be29da9edd8ceb7846dbdb49dd0da2c50cda004103d597 not found: ID does not exist"
Oct 30 01:22:36.973321 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.973294    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2"]
Oct 30 01:22:36.976485 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.976464    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38b47d-mfmx2"]
Oct 30 01:22:36.991545 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:36.991524    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"]
Oct 30 01:22:37.000589 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:37.000562    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-38b47d-6458f68b88-fd94q"]
Oct 30 01:22:38.263337 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:38.263304    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1272be00-9aa6-4cb1-9d1e-74cfe0164569" path="/var/lib/kubelet/pods/1272be00-9aa6-4cb1-9d1e-74cfe0164569/volumes"
Oct 30 01:22:38.263827 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:38.263810    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e87843f3-153e-4fc8-9771-316dfd0d2668" path="/var/lib/kubelet/pods/e87843f3-153e-4fc8-9771-316dfd0d2668/volumes"
Oct 30 01:22:39.442054 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.442010    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr" podUID="354c452d-1616-4d33-9d18-a44f6fc76366" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:22:39.442444 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.442055    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt" podUID="61780fe7-50f0-4f30-989c-03e165d46adf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:22:39.739748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.739640    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw"]
Oct 30 01:22:39.740291 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.740277    6099 cpu_manager.go:414] "containerMap: removing container" podUID="1272be00-9aa6-4cb1-9d1e-74cfe0164569" containerName="maas-api"
Oct 30 01:22:39.740338 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.740292    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1272be00-9aa6-4cb1-9d1e-74cfe0164569" containerName="maas-api"
Oct 30 01:22:39.740380 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.740342    6099 cpu_manager.go:414] "containerMap: removing container" podUID="8f79a646-0479-486f-8b0b-dda8e7ed0ac4" containerName="revoke-keys"
Oct 30 01:22:39.740380 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.740347    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f79a646-0479-486f-8b0b-dda8e7ed0ac4" containerName="revoke-keys"
Oct 30 01:22:39.740456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.740446    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1272be00-9aa6-4cb1-9d1e-74cfe0164569" containerName="maas-api"
Oct 30 01:22:39.740492 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.740466    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f79a646-0479-486f-8b0b-dda8e7ed0ac4" containerName="revoke-keys"
Oct 30 01:22:39.745622 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.745600    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw"
Oct 30 01:22:39.745829 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.745801    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt"]
Oct 30 01:22:39.749972 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.749949    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt"
Oct 30 01:22:39.761258 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.761242    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:22:39.761423 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.761404    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:22:39.776608 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.776578    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw"]
Oct 30 01:22:39.780362 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.780331    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt"]
Oct 30 01:22:39.839697 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.839669    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6086b2-4x5jw\" (UID: \"e774fee5-42fc-42c9-b02c-6b1cad0f80a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw"
Oct 30 01:22:39.839882 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.839845    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z86n\" (UniqueName: \"kubernetes.io/projected/ad0363d5-bc18-418f-8974-4d783829a6f4-kube-api-access-2z86n\") pod \"maas-api-revoke-sub-fcad322b-99nxt\" (UID: \"ad0363d5-bc18-418f-8974-4d783829a6f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt"
Oct 30 01:22:39.839882 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.839868    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6086b2-4x5jw\" (UID: \"e774fee5-42fc-42c9-b02c-6b1cad0f80a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw"
Oct 30 01:22:39.840007 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.839889    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0363d5-bc18-418f-8974-4d783829a6f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcad322b-99nxt\" (UID: \"ad0363d5-bc18-418f-8974-4d783829a6f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt"
Oct 30 01:22:39.840070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.840034    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0363d5-bc18-418f-8974-4d783829a6f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcad322b-99nxt\" (UID: \"ad0363d5-bc18-418f-8974-4d783829a6f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt"
Oct 30 01:22:39.840128 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.840074    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r6vr\" (UniqueName: \"kubernetes.io/projected/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-kube-api-access-2r6vr\") pod \"maas-api-revoke-sub-9d6086b2-4x5jw\" (UID: \"e774fee5-42fc-42c9-b02c-6b1cad0f80a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw"
Oct 30 01:22:39.941000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.940969    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z86n\" (UniqueName: \"kubernetes.io/projected/ad0363d5-bc18-418f-8974-4d783829a6f4-kube-api-access-2z86n\") pod \"maas-api-revoke-sub-fcad322b-99nxt\" (UID: \"ad0363d5-bc18-418f-8974-4d783829a6f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt"
Oct 30 01:22:39.941187 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.941011    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6086b2-4x5jw\" (UID: \"e774fee5-42fc-42c9-b02c-6b1cad0f80a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw"
Oct 30 01:22:39.941187 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.941035    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0363d5-bc18-418f-8974-4d783829a6f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcad322b-99nxt\" (UID: \"ad0363d5-bc18-418f-8974-4d783829a6f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt"
Oct 30 01:22:39.941187 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.941093    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0363d5-bc18-418f-8974-4d783829a6f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcad322b-99nxt\" (UID: \"ad0363d5-bc18-418f-8974-4d783829a6f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt"
Oct 30 01:22:39.941351 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.941274    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r6vr\" (UniqueName: \"kubernetes.io/projected/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-kube-api-access-2r6vr\") pod \"maas-api-revoke-sub-9d6086b2-4x5jw\" (UID: \"e774fee5-42fc-42c9-b02c-6b1cad0f80a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw"
Oct 30 01:22:39.941581 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.941490    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6086b2-4x5jw\" (UID: \"e774fee5-42fc-42c9-b02c-6b1cad0f80a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw"
Oct 30 01:22:39.941888 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.941852    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0363d5-bc18-418f-8974-4d783829a6f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcad322b-99nxt\" (UID: \"ad0363d5-bc18-418f-8974-4d783829a6f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt"
Oct 30 01:22:39.942163 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.942144    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6086b2-4x5jw\" (UID: \"e774fee5-42fc-42c9-b02c-6b1cad0f80a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw"
Oct 30 01:22:39.943605 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.943558    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0363d5-bc18-418f-8974-4d783829a6f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcad322b-99nxt\" (UID: \"ad0363d5-bc18-418f-8974-4d783829a6f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt"
Oct 30 01:22:39.943605 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.943576    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6086b2-4x5jw\" (UID: \"e774fee5-42fc-42c9-b02c-6b1cad0f80a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw"
Oct 30 01:22:39.964491 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.964451    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z86n\" (UniqueName: \"kubernetes.io/projected/ad0363d5-bc18-418f-8974-4d783829a6f4-kube-api-access-2z86n\") pod \"maas-api-revoke-sub-fcad322b-99nxt\" (UID: \"ad0363d5-bc18-418f-8974-4d783829a6f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt"
Oct 30 01:22:39.964491 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:39.964461    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r6vr\" (UniqueName: \"kubernetes.io/projected/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-kube-api-access-2r6vr\") pod \"maas-api-revoke-sub-9d6086b2-4x5jw\" (UID: \"e774fee5-42fc-42c9-b02c-6b1cad0f80a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw"
Oct 30 01:22:40.061984 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:40.061954    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw"
Oct 30 01:22:40.068075 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:40.068053    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt"
Oct 30 01:22:40.378166 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:40.378143    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw"]
Oct 30 01:22:40.378651 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:22:40.378632    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode774fee5_42fc_42c9_b02c_6b1cad0f80a8.slice/crio-9e2d9788a96306953496fca53376166f510b55aca8b7bbb491b5471a2900ce82 WatchSource:0}: Error finding container 9e2d9788a96306953496fca53376166f510b55aca8b7bbb491b5471a2900ce82: Status 404 returned error can't find the container with id 9e2d9788a96306953496fca53376166f510b55aca8b7bbb491b5471a2900ce82
Oct 30 01:22:40.391651 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:40.391624    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt"]
Oct 30 01:22:40.392613 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:22:40.392591    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0363d5_bc18_418f_8974_4d783829a6f4.slice/crio-e14111fb768586520e831afad0ff3269f4ffb9599a19bb006dc454422e999aa2 WatchSource:0}: Error finding container e14111fb768586520e831afad0ff3269f4ffb9599a19bb006dc454422e999aa2: Status 404 returned error can't find the container with id e14111fb768586520e831afad0ff3269f4ffb9599a19bb006dc454422e999aa2
Oct 30 01:22:40.969578 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:40.969534    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt" event={"ID":"ad0363d5-bc18-418f-8974-4d783829a6f4","Type":"ContainerStarted","Data":"d188440e0abafffa1bd8dbf43a678fed7c7aa3b9703580bfbf94b002bc4ed4a8"}
Oct 30 01:22:40.970069 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:40.969601    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt" event={"ID":"ad0363d5-bc18-418f-8974-4d783829a6f4","Type":"ContainerStarted","Data":"e14111fb768586520e831afad0ff3269f4ffb9599a19bb006dc454422e999aa2"}
Oct 30 01:22:40.971238 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:40.971210    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw" event={"ID":"e774fee5-42fc-42c9-b02c-6b1cad0f80a8","Type":"ContainerStarted","Data":"c7d2cc1e91152395fcfade760fe19629ac9a4c103d99d13c9c441247bcfefea8"}
Oct 30 01:22:40.971339 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:40.971259    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw" event={"ID":"e774fee5-42fc-42c9-b02c-6b1cad0f80a8","Type":"ContainerStarted","Data":"9e2d9788a96306953496fca53376166f510b55aca8b7bbb491b5471a2900ce82"}
Oct 30 01:22:41.017260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:41.017203    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt" podStartSLOduration=2.017186092 podStartE2EDuration="2.017186092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:41.016657128 +0000 UTC m=+2854.873564886" watchObservedRunningTime="2026-10-30 01:22:41.017186092 +0000 UTC m=+2854.874093848"
Oct 30 01:22:41.066353 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:41.066298    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw" podStartSLOduration=2.06628517 podStartE2EDuration="2.06628517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:41.06528041 +0000 UTC m=+2854.922188168" watchObservedRunningTime="2026-10-30 01:22:41.06628517 +0000 UTC m=+2854.923192936"
Oct 30 01:22:41.981091 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:41.981056    6099 generic.go:356] "Generic (PLEG): container finished" podID="ad0363d5-bc18-418f-8974-4d783829a6f4" containerID="d188440e0abafffa1bd8dbf43a678fed7c7aa3b9703580bfbf94b002bc4ed4a8" exitCode=0
Oct 30 01:22:41.981535 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:41.981140    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt" event={"ID":"ad0363d5-bc18-418f-8974-4d783829a6f4","Type":"ContainerDied","Data":"d188440e0abafffa1bd8dbf43a678fed7c7aa3b9703580bfbf94b002bc4ed4a8"}
Oct 30 01:22:41.983041 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:41.983003    6099 generic.go:356] "Generic (PLEG): container finished" podID="e774fee5-42fc-42c9-b02c-6b1cad0f80a8" containerID="c7d2cc1e91152395fcfade760fe19629ac9a4c103d99d13c9c441247bcfefea8" exitCode=0
Oct 30 01:22:41.983041 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:41.983032    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw" event={"ID":"e774fee5-42fc-42c9-b02c-6b1cad0f80a8","Type":"ContainerDied","Data":"c7d2cc1e91152395fcfade760fe19629ac9a4c103d99d13c9c441247bcfefea8"}
Oct 30 01:22:42.946692 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:42.946662    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt"
Oct 30 01:22:42.994035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:42.994007    6099 generic.go:356] "Generic (PLEG): container finished" podID="354c452d-1616-4d33-9d18-a44f6fc76366" containerID="030ee7d7bc1dfd3d6150d889b785b249161e12d8010c6903639a51550b636745" exitCode=0
Oct 30 01:22:42.994478 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:42.994132    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr" event={"ID":"354c452d-1616-4d33-9d18-a44f6fc76366","Type":"ContainerDied","Data":"030ee7d7bc1dfd3d6150d889b785b249161e12d8010c6903639a51550b636745"}
Oct 30 01:22:42.994478 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:42.994189    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr" event={"ID":"354c452d-1616-4d33-9d18-a44f6fc76366","Type":"ContainerDied","Data":"bebc199a9d92e2840c8cedc4d030b0965d28484ec74ceb4b760babd8c05f73d6"}
Oct 30 01:22:42.994478 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:42.994215    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bebc199a9d92e2840c8cedc4d030b0965d28484ec74ceb4b760babd8c05f73d6"
Oct 30 01:22:42.996608 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:42.996582    6099 generic.go:356] "Generic (PLEG): container finished" podID="61780fe7-50f0-4f30-989c-03e165d46adf" containerID="e813d54d5bf107aa6a67f4c8b4179365ea39b8456237216e49c85027e710b8bb" exitCode=0
Oct 30 01:22:42.996795 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:42.996658    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt"
Oct 30 01:22:42.996795 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:42.996673    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt" event={"ID":"61780fe7-50f0-4f30-989c-03e165d46adf","Type":"ContainerDied","Data":"e813d54d5bf107aa6a67f4c8b4179365ea39b8456237216e49c85027e710b8bb"}
Oct 30 01:22:42.996795 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:42.996771    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt" event={"ID":"61780fe7-50f0-4f30-989c-03e165d46adf","Type":"ContainerDied","Data":"63f90e014ca73b2c08b3e3a36edc574fe916d56f5920ef685258ba102fbd933d"}
Oct 30 01:22:42.996995 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:42.996816    6099 scope.go:122] "RemoveContainer" containerID="e813d54d5bf107aa6a67f4c8b4179365ea39b8456237216e49c85027e710b8bb"
Oct 30 01:22:43.006025 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.006000    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr"
Oct 30 01:22:43.020862 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.020836    6099 scope.go:122] "RemoveContainer" containerID="e813d54d5bf107aa6a67f4c8b4179365ea39b8456237216e49c85027e710b8bb"
Oct 30 01:22:43.021386 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:22:43.021358    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e813d54d5bf107aa6a67f4c8b4179365ea39b8456237216e49c85027e710b8bb\": container with ID starting with e813d54d5bf107aa6a67f4c8b4179365ea39b8456237216e49c85027e710b8bb not found: ID does not exist" containerID="e813d54d5bf107aa6a67f4c8b4179365ea39b8456237216e49c85027e710b8bb"
Oct 30 01:22:43.021594 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.021392    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e813d54d5bf107aa6a67f4c8b4179365ea39b8456237216e49c85027e710b8bb"} err="failed to get container status \"e813d54d5bf107aa6a67f4c8b4179365ea39b8456237216e49c85027e710b8bb\": rpc error: code = NotFound desc = could not find container \"e813d54d5bf107aa6a67f4c8b4179365ea39b8456237216e49c85027e710b8bb\": container with ID starting with e813d54d5bf107aa6a67f4c8b4179365ea39b8456237216e49c85027e710b8bb not found: ID does not exist"
Oct 30 01:22:43.057752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.056206    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61780fe7-50f0-4f30-989c-03e165d46adf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61780fe7-50f0-4f30-989c-03e165d46adf-plugins-config-volume\") pod \"61780fe7-50f0-4f30-989c-03e165d46adf\" (UID: \"61780fe7-50f0-4f30-989c-03e165d46adf\") "
Oct 30 01:22:43.057752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.056322    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/354c452d-1616-4d33-9d18-a44f6fc76366-kube-api-access-5n6mf\" (UniqueName: \"kubernetes.io/projected/354c452d-1616-4d33-9d18-a44f6fc76366-kube-api-access-5n6mf\") pod \"354c452d-1616-4d33-9d18-a44f6fc76366\" (UID: \"354c452d-1616-4d33-9d18-a44f6fc76366\") "
Oct 30 01:22:43.057752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.056363    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61780fe7-50f0-4f30-989c-03e165d46adf-kube-api-access-h479l\" (UniqueName: \"kubernetes.io/projected/61780fe7-50f0-4f30-989c-03e165d46adf-kube-api-access-h479l\") pod \"61780fe7-50f0-4f30-989c-03e165d46adf\" (UID: \"61780fe7-50f0-4f30-989c-03e165d46adf\") "
Oct 30 01:22:43.057752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.056520    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/354c452d-1616-4d33-9d18-a44f6fc76366-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354c452d-1616-4d33-9d18-a44f6fc76366-plugins-config-volume\") pod \"354c452d-1616-4d33-9d18-a44f6fc76366\" (UID: \"354c452d-1616-4d33-9d18-a44f6fc76366\") "
Oct 30 01:22:43.058478 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.058436    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354c452d-1616-4d33-9d18-a44f6fc76366-plugins-config-volume" pod "354c452d-1616-4d33-9d18-a44f6fc76366" (UID: "354c452d-1616-4d33-9d18-a44f6fc76366"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:43.058632 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.058608    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354c452d-1616-4d33-9d18-a44f6fc76366-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:43.058901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.058866    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61780fe7-50f0-4f30-989c-03e165d46adf-plugins-config-volume" pod "61780fe7-50f0-4f30-989c-03e165d46adf" (UID: "61780fe7-50f0-4f30-989c-03e165d46adf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:43.068876 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.068297    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354c452d-1616-4d33-9d18-a44f6fc76366-kube-api-access-5n6mf" pod "354c452d-1616-4d33-9d18-a44f6fc76366" (UID: "354c452d-1616-4d33-9d18-a44f6fc76366"). InnerVolumeSpecName "kube-api-access-5n6mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:43.073488 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.073437    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61780fe7-50f0-4f30-989c-03e165d46adf-kube-api-access-h479l" pod "61780fe7-50f0-4f30-989c-03e165d46adf" (UID: "61780fe7-50f0-4f30-989c-03e165d46adf"). InnerVolumeSpecName "kube-api-access-h479l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:43.199787 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.178973    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61780fe7-50f0-4f30-989c-03e165d46adf-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:43.199787 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.179014    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n6mf\" (UniqueName: \"kubernetes.io/projected/354c452d-1616-4d33-9d18-a44f6fc76366-kube-api-access-5n6mf\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:43.199787 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.179029    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h479l\" (UniqueName: \"kubernetes.io/projected/61780fe7-50f0-4f30-989c-03e165d46adf-kube-api-access-h479l\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:43.280425 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.280147    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt"
Oct 30 01:22:43.333084 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.332994    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw"
Oct 30 01:22:43.361175 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.361136    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt"]
Oct 30 01:22:43.369177 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.369150    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2b4105-85f86769d6-jqfkt"]
Oct 30 01:22:43.381550 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.381526    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-maas-api-xks-ca\") pod \"e774fee5-42fc-42c9-b02c-6b1cad0f80a8\" (UID: \"e774fee5-42fc-42c9-b02c-6b1cad0f80a8\") "
Oct 30 01:22:43.381691 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.381678    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0363d5-bc18-418f-8974-4d783829a6f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0363d5-bc18-418f-8974-4d783829a6f4-maas-api-openshift-ca\") pod \"ad0363d5-bc18-418f-8974-4d783829a6f4\" (UID: \"ad0363d5-bc18-418f-8974-4d783829a6f4\") "
Oct 30 01:22:43.381777 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.381701    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0363d5-bc18-418f-8974-4d783829a6f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0363d5-bc18-418f-8974-4d783829a6f4-maas-api-xks-ca\") pod \"ad0363d5-bc18-418f-8974-4d783829a6f4\" (UID: \"ad0363d5-bc18-418f-8974-4d783829a6f4\") "
Oct 30 01:22:43.381777 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.381766    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-maas-api-openshift-ca\") pod \"e774fee5-42fc-42c9-b02c-6b1cad0f80a8\" (UID: \"e774fee5-42fc-42c9-b02c-6b1cad0f80a8\") "
Oct 30 01:22:43.381970 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.381814    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-kube-api-access-2r6vr\" (UniqueName: \"kubernetes.io/projected/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-kube-api-access-2r6vr\") pod \"e774fee5-42fc-42c9-b02c-6b1cad0f80a8\" (UID: \"e774fee5-42fc-42c9-b02c-6b1cad0f80a8\") "
Oct 30 01:22:43.381970 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.381902    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0363d5-bc18-418f-8974-4d783829a6f4-kube-api-access-2z86n\" (UniqueName: \"kubernetes.io/projected/ad0363d5-bc18-418f-8974-4d783829a6f4-kube-api-access-2z86n\") pod \"ad0363d5-bc18-418f-8974-4d783829a6f4\" (UID: \"ad0363d5-bc18-418f-8974-4d783829a6f4\") "
Oct 30 01:22:43.382140 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.382112    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0363d5-bc18-418f-8974-4d783829a6f4-maas-api-openshift-ca" pod "ad0363d5-bc18-418f-8974-4d783829a6f4" (UID: "ad0363d5-bc18-418f-8974-4d783829a6f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:43.382653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.382575    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0363d5-bc18-418f-8974-4d783829a6f4-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:43.382653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.382618    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-maas-api-openshift-ca" pod "e774fee5-42fc-42c9-b02c-6b1cad0f80a8" (UID: "e774fee5-42fc-42c9-b02c-6b1cad0f80a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:43.385740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.385654    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0363d5-bc18-418f-8974-4d783829a6f4-kube-api-access-2z86n" pod "ad0363d5-bc18-418f-8974-4d783829a6f4" (UID: "ad0363d5-bc18-418f-8974-4d783829a6f4"). InnerVolumeSpecName "kube-api-access-2z86n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:43.392271 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.391741    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-kube-api-access-2r6vr" pod "e774fee5-42fc-42c9-b02c-6b1cad0f80a8" (UID: "e774fee5-42fc-42c9-b02c-6b1cad0f80a8"). InnerVolumeSpecName "kube-api-access-2r6vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:43.392271 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.392018    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-maas-api-xks-ca" pod "e774fee5-42fc-42c9-b02c-6b1cad0f80a8" (UID: "e774fee5-42fc-42c9-b02c-6b1cad0f80a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:43.393351 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.393327    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0363d5-bc18-418f-8974-4d783829a6f4-maas-api-xks-ca" pod "ad0363d5-bc18-418f-8974-4d783829a6f4" (UID: "ad0363d5-bc18-418f-8974-4d783829a6f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:43.484149 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.484075    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z86n\" (UniqueName: \"kubernetes.io/projected/ad0363d5-bc18-418f-8974-4d783829a6f4-kube-api-access-2z86n\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:43.484149 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.484102    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:43.484149 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.484112    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0363d5-bc18-418f-8974-4d783829a6f4-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:43.484149 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.484120    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:43.484149 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:43.484131    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r6vr\" (UniqueName: \"kubernetes.io/projected/e774fee5-42fc-42c9-b02c-6b1cad0f80a8-kube-api-access-2r6vr\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:44.004782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:44.004745    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw"
Oct 30 01:22:44.005236 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:44.004744    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw" event={"ID":"e774fee5-42fc-42c9-b02c-6b1cad0f80a8","Type":"ContainerDied","Data":"9e2d9788a96306953496fca53376166f510b55aca8b7bbb491b5471a2900ce82"}
Oct 30 01:22:44.005236 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:44.004909    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e2d9788a96306953496fca53376166f510b55aca8b7bbb491b5471a2900ce82"
Oct 30 01:22:44.007385 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:44.007364    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt"
Oct 30 01:22:44.007498 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:44.007403    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt" event={"ID":"ad0363d5-bc18-418f-8974-4d783829a6f4","Type":"ContainerDied","Data":"e14111fb768586520e831afad0ff3269f4ffb9599a19bb006dc454422e999aa2"}
Oct 30 01:22:44.007498 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:44.007443    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e14111fb768586520e831afad0ff3269f4ffb9599a19bb006dc454422e999aa2"
Oct 30 01:22:44.007579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:44.007544    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr"
Oct 30 01:22:44.098031 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:44.097999    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr"]
Oct 30 01:22:44.100992 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:44.100966    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2b4105-755d7d6c47-tvwdr"]
Oct 30 01:22:44.263064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:44.262991    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="354c452d-1616-4d33-9d18-a44f6fc76366" path="/var/lib/kubelet/pods/354c452d-1616-4d33-9d18-a44f6fc76366/volumes"
Oct 30 01:22:44.263623 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:44.263601    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61780fe7-50f0-4f30-989c-03e165d46adf" path="/var/lib/kubelet/pods/61780fe7-50f0-4f30-989c-03e165d46adf/volumes"
Oct 30 01:22:45.448688 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.448650    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"]
Oct 30 01:22:45.449118 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.448908    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp" podUID="51e427c9-5c89-4401-afa6-437676618dbf" containerName="istio-proxy" containerID="cri-o://4f6a2e2707560db95c8166f11273556c88045fc6624f92f41dc7043dcc5bea0f" gracePeriod=30
Oct 30 01:22:45.671407 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.671373    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m"]
Oct 30 01:22:45.672155 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.672128    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e774fee5-42fc-42c9-b02c-6b1cad0f80a8" containerName="revoke-keys"
Oct 30 01:22:45.672155 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.672150    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e774fee5-42fc-42c9-b02c-6b1cad0f80a8" containerName="revoke-keys"
Oct 30 01:22:45.672382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.672176    6099 cpu_manager.go:414] "containerMap: removing container" podUID="ad0363d5-bc18-418f-8974-4d783829a6f4" containerName="revoke-keys"
Oct 30 01:22:45.672382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.672181    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0363d5-bc18-418f-8974-4d783829a6f4" containerName="revoke-keys"
Oct 30 01:22:45.672382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.672199    6099 cpu_manager.go:414] "containerMap: removing container" podUID="61780fe7-50f0-4f30-989c-03e165d46adf" containerName="payload-processing"
Oct 30 01:22:45.672382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.672204    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61780fe7-50f0-4f30-989c-03e165d46adf" containerName="payload-processing"
Oct 30 01:22:45.672382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.672215    6099 cpu_manager.go:414] "containerMap: removing container" podUID="354c452d-1616-4d33-9d18-a44f6fc76366" containerName="payload-pre-processing"
Oct 30 01:22:45.672382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.672220    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="354c452d-1616-4d33-9d18-a44f6fc76366" containerName="payload-pre-processing"
Oct 30 01:22:45.672382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.672309    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e774fee5-42fc-42c9-b02c-6b1cad0f80a8" containerName="revoke-keys"
Oct 30 01:22:45.672382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.672319    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0363d5-bc18-418f-8974-4d783829a6f4" containerName="revoke-keys"
Oct 30 01:22:45.672382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.672329    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="61780fe7-50f0-4f30-989c-03e165d46adf" containerName="payload-processing"
Oct 30 01:22:45.672382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.672337    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="354c452d-1616-4d33-9d18-a44f6fc76366" containerName="payload-pre-processing"
Oct 30 01:22:45.678088 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.678062    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m"
Oct 30 01:22:45.693421 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.693382    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m"]
Oct 30 01:22:45.720905 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.720825    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116cd47b-60a5-4ca6-874e-f9366967066f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m\" (UID: \"116cd47b-60a5-4ca6-874e-f9366967066f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m"
Oct 30 01:22:45.720905 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.720893    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nbjh\" (UniqueName: \"kubernetes.io/projected/116cd47b-60a5-4ca6-874e-f9366967066f-kube-api-access-6nbjh\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m\" (UID: \"116cd47b-60a5-4ca6-874e-f9366967066f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m"
Oct 30 01:22:45.721110 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.721018    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116cd47b-60a5-4ca6-874e-f9366967066f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m\" (UID: \"116cd47b-60a5-4ca6-874e-f9366967066f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m"
Oct 30 01:22:45.759552 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.759524    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:22:45.759682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.759567    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:22:45.821981 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.821952    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116cd47b-60a5-4ca6-874e-f9366967066f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m\" (UID: \"116cd47b-60a5-4ca6-874e-f9366967066f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m"
Oct 30 01:22:45.822104 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.822084    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116cd47b-60a5-4ca6-874e-f9366967066f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m\" (UID: \"116cd47b-60a5-4ca6-874e-f9366967066f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m"
Oct 30 01:22:45.822147 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.822133    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nbjh\" (UniqueName: \"kubernetes.io/projected/116cd47b-60a5-4ca6-874e-f9366967066f-kube-api-access-6nbjh\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m\" (UID: \"116cd47b-60a5-4ca6-874e-f9366967066f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m"
Oct 30 01:22:45.822679 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.822656    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116cd47b-60a5-4ca6-874e-f9366967066f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m\" (UID: \"116cd47b-60a5-4ca6-874e-f9366967066f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m"
Oct 30 01:22:45.824312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.824290    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116cd47b-60a5-4ca6-874e-f9366967066f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m\" (UID: \"116cd47b-60a5-4ca6-874e-f9366967066f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m"
Oct 30 01:22:45.845796 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.845768    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nbjh\" (UniqueName: \"kubernetes.io/projected/116cd47b-60a5-4ca6-874e-f9366967066f-kube-api-access-6nbjh\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m\" (UID: \"116cd47b-60a5-4ca6-874e-f9366967066f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m"
Oct 30 01:22:45.976552 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.976479    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b"]
Oct 30 01:22:45.981305 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.981282    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b"
Oct 30 01:22:45.996667 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:45.996645    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m"
Oct 30 01:22:46.002321 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:46.001566    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b"]
Oct 30 01:22:46.025120 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:46.025088    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txsh8\" (UniqueName: \"kubernetes.io/projected/a2c329c9-52a2-47e9-9ba0-5d9021338e37-kube-api-access-txsh8\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b\" (UID: \"a2c329c9-52a2-47e9-9ba0-5d9021338e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b"
Oct 30 01:22:46.025613 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:46.025343    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c329c9-52a2-47e9-9ba0-5d9021338e37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b\" (UID: \"a2c329c9-52a2-47e9-9ba0-5d9021338e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b"
Oct 30 01:22:46.025613 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:46.025387    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c329c9-52a2-47e9-9ba0-5d9021338e37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b\" (UID: \"a2c329c9-52a2-47e9-9ba0-5d9021338e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b"
Oct 30 01:22:46.127063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:46.127033    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c329c9-52a2-47e9-9ba0-5d9021338e37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b\" (UID: \"a2c329c9-52a2-47e9-9ba0-5d9021338e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b"
Oct 30 01:22:46.127218 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:46.127070    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c329c9-52a2-47e9-9ba0-5d9021338e37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b\" (UID: \"a2c329c9-52a2-47e9-9ba0-5d9021338e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b"
Oct 30 01:22:46.127765 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:46.127234    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txsh8\" (UniqueName: \"kubernetes.io/projected/a2c329c9-52a2-47e9-9ba0-5d9021338e37-kube-api-access-txsh8\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b\" (UID: \"a2c329c9-52a2-47e9-9ba0-5d9021338e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b"
Oct 30 01:22:46.127857 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:46.127837    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c329c9-52a2-47e9-9ba0-5d9021338e37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b\" (UID: \"a2c329c9-52a2-47e9-9ba0-5d9021338e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b"
Oct 30 01:22:46.129441 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:46.129420    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c329c9-52a2-47e9-9ba0-5d9021338e37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b\" (UID: \"a2c329c9-52a2-47e9-9ba0-5d9021338e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b"
Oct 30 01:22:46.152308 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:46.152281    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txsh8\" (UniqueName: \"kubernetes.io/projected/a2c329c9-52a2-47e9-9ba0-5d9021338e37-kube-api-access-txsh8\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b\" (UID: \"a2c329c9-52a2-47e9-9ba0-5d9021338e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b"
Oct 30 01:22:46.296717 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:46.296685    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b"
Oct 30 01:22:46.320447 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:22:46.320407    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116cd47b_60a5_4ca6_874e_f9366967066f.slice/crio-dadcc806b1d613f90c461c2cadc4f99f17a893daef0f08d7b24a984887518a9d WatchSource:0}: Error finding container dadcc806b1d613f90c461c2cadc4f99f17a893daef0f08d7b24a984887518a9d: Status 404 returned error can't find the container with id dadcc806b1d613f90c461c2cadc4f99f17a893daef0f08d7b24a984887518a9d
Oct 30 01:22:46.320643 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:46.320626    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m"]
Oct 30 01:22:46.601368 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:22:46.601339    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c329c9_52a2_47e9_9ba0_5d9021338e37.slice/crio-60490f3060cf3e89c98191e2acf3d929074a533978aa788d5f4285a1ea7451b9 WatchSource:0}: Error finding container 60490f3060cf3e89c98191e2acf3d929074a533978aa788d5f4285a1ea7451b9: Status 404 returned error can't find the container with id 60490f3060cf3e89c98191e2acf3d929074a533978aa788d5f4285a1ea7451b9
Oct 30 01:22:46.601980 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:46.601910    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b"]
Oct 30 01:22:46.630816 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:22:46.630784    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:22:47.035894 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:47.035855    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m" event={"ID":"116cd47b-60a5-4ca6-874e-f9366967066f","Type":"ContainerStarted","Data":"ea4e05531671949b56d171ff90180c41a9e221e648926d3df3fc6b8f02ef4d03"}
Oct 30 01:22:47.036086 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:47.035917    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m" event={"ID":"116cd47b-60a5-4ca6-874e-f9366967066f","Type":"ContainerStarted","Data":"dadcc806b1d613f90c461c2cadc4f99f17a893daef0f08d7b24a984887518a9d"}
Oct 30 01:22:47.037689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:47.037661    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b" event={"ID":"a2c329c9-52a2-47e9-9ba0-5d9021338e37","Type":"ContainerStarted","Data":"48a909c8ae54b945ddb3b66815cf81bb0673316162271f395d454cbee780038b"}
Oct 30 01:22:47.037845 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:47.037723    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b" event={"ID":"a2c329c9-52a2-47e9-9ba0-5d9021338e37","Type":"ContainerStarted","Data":"60490f3060cf3e89c98191e2acf3d929074a533978aa788d5f4285a1ea7451b9"}
Oct 30 01:22:47.085428 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:47.085378    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m" podStartSLOduration=2.085364856 podStartE2EDuration="2.085364856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:47.083314372 +0000 UTC m=+2860.940222127" watchObservedRunningTime="2026-10-30 01:22:47.085364856 +0000 UTC m=+2860.942272611"
Oct 30 01:22:47.142681 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:47.142596    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b" podStartSLOduration=2.142581833 podStartE2EDuration="2.142581833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:47.139847073 +0000 UTC m=+2860.996754828" watchObservedRunningTime="2026-10-30 01:22:47.142581833 +0000 UTC m=+2860.999489588"
Oct 30 01:22:48.047814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:48.047761    6099 generic.go:356] "Generic (PLEG): container finished" podID="a2c329c9-52a2-47e9-9ba0-5d9021338e37" containerID="48a909c8ae54b945ddb3b66815cf81bb0673316162271f395d454cbee780038b" exitCode=0
Oct 30 01:22:48.048294 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:48.047840    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b" event={"ID":"a2c329c9-52a2-47e9-9ba0-5d9021338e37","Type":"ContainerDied","Data":"48a909c8ae54b945ddb3b66815cf81bb0673316162271f395d454cbee780038b"}
Oct 30 01:22:48.049684 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:48.049656    6099 generic.go:356] "Generic (PLEG): container finished" podID="116cd47b-60a5-4ca6-874e-f9366967066f" containerID="ea4e05531671949b56d171ff90180c41a9e221e648926d3df3fc6b8f02ef4d03" exitCode=0
Oct 30 01:22:48.049835 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:48.049694    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m" event={"ID":"116cd47b-60a5-4ca6-874e-f9366967066f","Type":"ContainerDied","Data":"ea4e05531671949b56d171ff90180c41a9e221e648926d3df3fc6b8f02ef4d03"}
Oct 30 01:22:49.274797 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.274765    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m"
Oct 30 01:22:49.285824 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.285795    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b"
Oct 30 01:22:49.380818 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.380740    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c329c9-52a2-47e9-9ba0-5d9021338e37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c329c9-52a2-47e9-9ba0-5d9021338e37-maas-api-openshift-ca\") pod \"a2c329c9-52a2-47e9-9ba0-5d9021338e37\" (UID: \"a2c329c9-52a2-47e9-9ba0-5d9021338e37\") "
Oct 30 01:22:49.380979 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.380837    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2c329c9-52a2-47e9-9ba0-5d9021338e37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c329c9-52a2-47e9-9ba0-5d9021338e37-maas-api-xks-ca\") pod \"a2c329c9-52a2-47e9-9ba0-5d9021338e37\" (UID: \"a2c329c9-52a2-47e9-9ba0-5d9021338e37\") "
Oct 30 01:22:49.380979 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.380892    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/116cd47b-60a5-4ca6-874e-f9366967066f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116cd47b-60a5-4ca6-874e-f9366967066f-maas-api-openshift-ca\") pod \"116cd47b-60a5-4ca6-874e-f9366967066f\" (UID: \"116cd47b-60a5-4ca6-874e-f9366967066f\") "
Oct 30 01:22:49.380979 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.380939    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/116cd47b-60a5-4ca6-874e-f9366967066f-kube-api-access-6nbjh\" (UniqueName: \"kubernetes.io/projected/116cd47b-60a5-4ca6-874e-f9366967066f-kube-api-access-6nbjh\") pod \"116cd47b-60a5-4ca6-874e-f9366967066f\" (UID: \"116cd47b-60a5-4ca6-874e-f9366967066f\") "
Oct 30 01:22:49.381179 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.381095    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c329c9-52a2-47e9-9ba0-5d9021338e37-maas-api-openshift-ca" pod "a2c329c9-52a2-47e9-9ba0-5d9021338e37" (UID: "a2c329c9-52a2-47e9-9ba0-5d9021338e37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:49.381311 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.381280    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c329c9-52a2-47e9-9ba0-5d9021338e37-kube-api-access-txsh8\" (UniqueName: \"kubernetes.io/projected/a2c329c9-52a2-47e9-9ba0-5d9021338e37-kube-api-access-txsh8\") pod \"a2c329c9-52a2-47e9-9ba0-5d9021338e37\" (UID: \"a2c329c9-52a2-47e9-9ba0-5d9021338e37\") "
Oct 30 01:22:49.381450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.381330    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/116cd47b-60a5-4ca6-874e-f9366967066f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116cd47b-60a5-4ca6-874e-f9366967066f-maas-api-xks-ca\") pod \"116cd47b-60a5-4ca6-874e-f9366967066f\" (UID: \"116cd47b-60a5-4ca6-874e-f9366967066f\") "
Oct 30 01:22:49.381450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.381331    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/116cd47b-60a5-4ca6-874e-f9366967066f-maas-api-openshift-ca" pod "116cd47b-60a5-4ca6-874e-f9366967066f" (UID: "116cd47b-60a5-4ca6-874e-f9366967066f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:49.382612 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.382578    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c329c9-52a2-47e9-9ba0-5d9021338e37-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:49.382881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.382835    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116cd47b-60a5-4ca6-874e-f9366967066f-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:49.384606 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.384570    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c329c9-52a2-47e9-9ba0-5d9021338e37-kube-api-access-txsh8" pod "a2c329c9-52a2-47e9-9ba0-5d9021338e37" (UID: "a2c329c9-52a2-47e9-9ba0-5d9021338e37"). InnerVolumeSpecName "kube-api-access-txsh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:49.384855 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.384830    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/116cd47b-60a5-4ca6-874e-f9366967066f-maas-api-xks-ca" pod "116cd47b-60a5-4ca6-874e-f9366967066f" (UID: "116cd47b-60a5-4ca6-874e-f9366967066f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:49.385088 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.385064    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c329c9-52a2-47e9-9ba0-5d9021338e37-maas-api-xks-ca" pod "a2c329c9-52a2-47e9-9ba0-5d9021338e37" (UID: "a2c329c9-52a2-47e9-9ba0-5d9021338e37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:49.386750 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.386691    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116cd47b-60a5-4ca6-874e-f9366967066f-kube-api-access-6nbjh" pod "116cd47b-60a5-4ca6-874e-f9366967066f" (UID: "116cd47b-60a5-4ca6-874e-f9366967066f"). InnerVolumeSpecName "kube-api-access-6nbjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:49.483822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.483788    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txsh8\" (UniqueName: \"kubernetes.io/projected/a2c329c9-52a2-47e9-9ba0-5d9021338e37-kube-api-access-txsh8\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:49.483822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.483817    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116cd47b-60a5-4ca6-874e-f9366967066f-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:49.483822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.483827    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c329c9-52a2-47e9-9ba0-5d9021338e37-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:49.484063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:49.483836    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nbjh\" (UniqueName: \"kubernetes.io/projected/116cd47b-60a5-4ca6-874e-f9366967066f-kube-api-access-6nbjh\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:50.065981 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.065951    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b"
Oct 30 01:22:50.066190 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.065989    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b" event={"ID":"a2c329c9-52a2-47e9-9ba0-5d9021338e37","Type":"ContainerDied","Data":"60490f3060cf3e89c98191e2acf3d929074a533978aa788d5f4285a1ea7451b9"}
Oct 30 01:22:50.066190 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.066046    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60490f3060cf3e89c98191e2acf3d929074a533978aa788d5f4285a1ea7451b9"
Oct 30 01:22:50.067590 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.067563    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m"
Oct 30 01:22:50.067741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.067609    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m" event={"ID":"116cd47b-60a5-4ca6-874e-f9366967066f","Type":"ContainerDied","Data":"dadcc806b1d613f90c461c2cadc4f99f17a893daef0f08d7b24a984887518a9d"}
Oct 30 01:22:50.067741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.067653    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dadcc806b1d613f90c461c2cadc4f99f17a893daef0f08d7b24a984887518a9d"
Oct 30 01:22:50.723090 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.723034    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:22:50.802130 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.802091    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-workload-certs\") pod \"51e427c9-5c89-4401-afa6-437676618dbf\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") "
Oct 30 01:22:50.802307 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.802162    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-workload-socket\") pod \"51e427c9-5c89-4401-afa6-437676618dbf\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") "
Oct 30 01:22:50.802307 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.802223    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-istio-data\") pod \"51e427c9-5c89-4401-afa6-437676618dbf\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") "
Oct 30 01:22:50.802307 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.802290    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e427c9-5c89-4401-afa6-437676618dbf-kube-api-access-r77d4\" (UniqueName: \"kubernetes.io/projected/51e427c9-5c89-4401-afa6-437676618dbf-kube-api-access-r77d4\") pod \"51e427c9-5c89-4401-afa6-437676618dbf\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") "
Oct 30 01:22:50.802548 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.802381    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51e427c9-5c89-4401-afa6-437676618dbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e427c9-5c89-4401-afa6-437676618dbf-istio-podinfo\") pod \"51e427c9-5c89-4401-afa6-437676618dbf\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") "
Oct 30 01:22:50.802548 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.802385    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-workload-certs" pod "51e427c9-5c89-4401-afa6-437676618dbf" (UID: "51e427c9-5c89-4401-afa6-437676618dbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:50.802548 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.802518    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-workload-socket" pod "51e427c9-5c89-4401-afa6-437676618dbf" (UID: "51e427c9-5c89-4401-afa6-437676618dbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:50.802752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.802549    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e427c9-5c89-4401-afa6-437676618dbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e427c9-5c89-4401-afa6-437676618dbf-istiod-ca-cert\") pod \"51e427c9-5c89-4401-afa6-437676618dbf\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") "
Oct 30 01:22:50.802752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.802658    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e427c9-5c89-4401-afa6-437676618dbf-istio-token\" (UniqueName: \"kubernetes.io/projected/51e427c9-5c89-4401-afa6-437676618dbf-istio-token\") pod \"51e427c9-5c89-4401-afa6-437676618dbf\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") "
Oct 30 01:22:50.802752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.802689    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-istio-envoy\") pod \"51e427c9-5c89-4401-afa6-437676618dbf\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") "
Oct 30 01:22:50.802752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.802747    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-credential-socket\") pod \"51e427c9-5c89-4401-afa6-437676618dbf\" (UID: \"51e427c9-5c89-4401-afa6-437676618dbf\") "
Oct 30 01:22:50.802995 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.802958    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e427c9-5c89-4401-afa6-437676618dbf-istiod-ca-cert" pod "51e427c9-5c89-4401-afa6-437676618dbf" (UID: "51e427c9-5c89-4401-afa6-437676618dbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:50.803243 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.803219    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-credential-socket" pod "51e427c9-5c89-4401-afa6-437676618dbf" (UID: "51e427c9-5c89-4401-afa6-437676618dbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:50.803583 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.803561    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e427c9-5c89-4401-afa6-437676618dbf-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:50.803656 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.803589    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:50.803656 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.803604    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:50.803656 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.803618    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:50.803860 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.803807    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-istio-data" pod "51e427c9-5c89-4401-afa6-437676618dbf" (UID: "51e427c9-5c89-4401-afa6-437676618dbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:50.805041 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.804958    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51e427c9-5c89-4401-afa6-437676618dbf-istio-podinfo" pod "51e427c9-5c89-4401-afa6-437676618dbf" (UID: "51e427c9-5c89-4401-afa6-437676618dbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:22:50.805468 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.805443    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e427c9-5c89-4401-afa6-437676618dbf-istio-token" pod "51e427c9-5c89-4401-afa6-437676618dbf" (UID: "51e427c9-5c89-4401-afa6-437676618dbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:50.805744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.805683    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e427c9-5c89-4401-afa6-437676618dbf-kube-api-access-r77d4" pod "51e427c9-5c89-4401-afa6-437676618dbf" (UID: "51e427c9-5c89-4401-afa6-437676618dbf"). InnerVolumeSpecName "kube-api-access-r77d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:50.805973 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.805946    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-istio-envoy" pod "51e427c9-5c89-4401-afa6-437676618dbf" (UID: "51e427c9-5c89-4401-afa6-437676618dbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:50.904830 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.904800    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:50.904830 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.904826    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r77d4\" (UniqueName: \"kubernetes.io/projected/51e427c9-5c89-4401-afa6-437676618dbf-kube-api-access-r77d4\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:50.904992 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.904842    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e427c9-5c89-4401-afa6-437676618dbf-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:50.904992 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.904855    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e427c9-5c89-4401-afa6-437676618dbf-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:50.904992 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:50.904867    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e427c9-5c89-4401-afa6-437676618dbf-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:22:51.084791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:51.084752    6099 generic.go:356] "Generic (PLEG): container finished" podID="51e427c9-5c89-4401-afa6-437676618dbf" containerID="4f6a2e2707560db95c8166f11273556c88045fc6624f92f41dc7043dcc5bea0f" exitCode=0
Oct 30 01:22:51.085023 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:51.084869    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp" event={"ID":"51e427c9-5c89-4401-afa6-437676618dbf","Type":"ContainerDied","Data":"4f6a2e2707560db95c8166f11273556c88045fc6624f92f41dc7043dcc5bea0f"}
Oct 30 01:22:51.085023 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:51.084937    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp" event={"ID":"51e427c9-5c89-4401-afa6-437676618dbf","Type":"ContainerDied","Data":"431a4081ca0e21f565a1d8a311b3db9533881b6114cbbe88b6e85c9f4b580c99"}
Oct 30 01:22:51.085023 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:51.084972    6099 scope.go:122] "RemoveContainer" containerID="4f6a2e2707560db95c8166f11273556c88045fc6624f92f41dc7043dcc5bea0f"
Oct 30 01:22:51.085274 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:51.085250    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"
Oct 30 01:22:51.105452 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:51.105429    6099 scope.go:122] "RemoveContainer" containerID="509c1ad84607be8d861219d828745769d199bccd3408f168903ad43a347af7d8"
Oct 30 01:22:51.116736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:51.116702    6099 scope.go:122] "RemoveContainer" containerID="4f6a2e2707560db95c8166f11273556c88045fc6624f92f41dc7043dcc5bea0f"
Oct 30 01:22:51.117045 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:22:51.117027    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f6a2e2707560db95c8166f11273556c88045fc6624f92f41dc7043dcc5bea0f\": container with ID starting with 4f6a2e2707560db95c8166f11273556c88045fc6624f92f41dc7043dcc5bea0f not found: ID does not exist" containerID="4f6a2e2707560db95c8166f11273556c88045fc6624f92f41dc7043dcc5bea0f"
Oct 30 01:22:51.117090 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:51.117054    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6a2e2707560db95c8166f11273556c88045fc6624f92f41dc7043dcc5bea0f"} err="failed to get container status \"4f6a2e2707560db95c8166f11273556c88045fc6624f92f41dc7043dcc5bea0f\": rpc error: code = NotFound desc = could not find container \"4f6a2e2707560db95c8166f11273556c88045fc6624f92f41dc7043dcc5bea0f\": container with ID starting with 4f6a2e2707560db95c8166f11273556c88045fc6624f92f41dc7043dcc5bea0f not found: ID does not exist"
Oct 30 01:22:51.117090 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:51.117074    6099 scope.go:122] "RemoveContainer" containerID="509c1ad84607be8d861219d828745769d199bccd3408f168903ad43a347af7d8"
Oct 30 01:22:51.117338 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:22:51.117318    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509c1ad84607be8d861219d828745769d199bccd3408f168903ad43a347af7d8\": container with ID starting with 509c1ad84607be8d861219d828745769d199bccd3408f168903ad43a347af7d8 not found: ID does not exist" containerID="509c1ad84607be8d861219d828745769d199bccd3408f168903ad43a347af7d8"
Oct 30 01:22:51.117384 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:51.117343    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509c1ad84607be8d861219d828745769d199bccd3408f168903ad43a347af7d8"} err="failed to get container status \"509c1ad84607be8d861219d828745769d199bccd3408f168903ad43a347af7d8\": rpc error: code = NotFound desc = could not find container \"509c1ad84607be8d861219d828745769d199bccd3408f168903ad43a347af7d8\": container with ID starting with 509c1ad84607be8d861219d828745769d199bccd3408f168903ad43a347af7d8 not found: ID does not exist"
Oct 30 01:22:51.140215 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:51.140177    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"]
Oct 30 01:22:51.144405 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:51.144381    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6cba49-openshift-default-6c65f96df5-7nrwp"]
Oct 30 01:22:52.262289 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:52.262259    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e427c9-5c89-4401-afa6-437676618dbf" path="/var/lib/kubelet/pods/51e427c9-5c89-4401-afa6-437676618dbf/volumes"
Oct 30 01:22:56.423847 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:56.423811    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq"]
Oct 30 01:22:56.454824 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:56.454790    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9"]
Oct 30 01:22:57.695918 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:57.695877    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m"]
Oct 30 01:22:57.698740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:57.698715    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-969b92-2st9m"]
Oct 30 01:22:58.263638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:22:58.263604    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="116cd47b-60a5-4ca6-874e-f9366967066f" path="/var/lib/kubelet/pods/116cd47b-60a5-4ca6-874e-f9366967066f/volumes"
Oct 30 01:23:01.424493 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:01.424448    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq" podUID="6adc9cb6-daf1-4c37-b49f-96cfaf6a8387" containerName="payload-processing" containerID="cri-o://29e1ae1ddaecfe7808b1edbab9ed6f73d8303ecb59bec59eff0839d46ada3a6d" gracePeriod=25
Oct 30 01:23:01.455797 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:01.455765    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9" podUID="6b90542e-141d-47e4-8270-4aa8e67632ca" containerName="payload-pre-processing" containerID="cri-o://a2fc4d297acb093a1d6c25691d94c54589074f387be1260ecbf063c85ec8f88d" gracePeriod=25
Oct 30 01:23:05.138992 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.138953    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b_261fed99-0c9e-42cb-ad91-540d2b125576/payload-processing/0.log"
Oct 30 01:23:05.246220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.246186    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"]
Oct 30 01:23:05.246448 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.246409    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x" podUID="ed244c34-4856-46d8-a180-030fd7d4362e" containerName="istio-proxy" containerID="cri-o://fd049efca35e876e1ce093c359eeb2f77c28ba75ab448e497f9b96c9b7dcc672" gracePeriod=30
Oct 30 01:23:05.403110 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.403017    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rcxc5_1569a420-ef8b-4ca9-86c1-cde78845b4b6/payload-processing/0.log"
Oct 30 01:23:05.612345 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.612319    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rcxc5_1569a420-ef8b-4ca9-86c1-cde78845b4b6/payload-processing/0.log"
Oct 30 01:23:05.803475 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.803439    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh"]
Oct 30 01:23:05.804086 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.804070    6099 cpu_manager.go:414] "containerMap: removing container" podUID="51e427c9-5c89-4401-afa6-437676618dbf" containerName="istio-proxy"
Oct 30 01:23:05.804138 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.804088    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e427c9-5c89-4401-afa6-437676618dbf" containerName="istio-proxy"
Oct 30 01:23:05.804138 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.804100    6099 cpu_manager.go:414] "containerMap: removing container" podUID="51e427c9-5c89-4401-afa6-437676618dbf" containerName="istio-proxy"
Oct 30 01:23:05.804138 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.804105    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e427c9-5c89-4401-afa6-437676618dbf" containerName="istio-proxy"
Oct 30 01:23:05.804138 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.804125    6099 cpu_manager.go:414] "containerMap: removing container" podUID="116cd47b-60a5-4ca6-874e-f9366967066f" containerName="revoke-keys"
Oct 30 01:23:05.804138 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.804130    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="116cd47b-60a5-4ca6-874e-f9366967066f" containerName="revoke-keys"
Oct 30 01:23:05.804311 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.804159    6099 cpu_manager.go:414] "containerMap: removing container" podUID="a2c329c9-52a2-47e9-9ba0-5d9021338e37" containerName="revoke-keys"
Oct 30 01:23:05.804311 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.804164    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c329c9-52a2-47e9-9ba0-5d9021338e37" containerName="revoke-keys"
Oct 30 01:23:05.808025 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.804814    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="116cd47b-60a5-4ca6-874e-f9366967066f" containerName="revoke-keys"
Oct 30 01:23:05.808025 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.804915    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c329c9-52a2-47e9-9ba0-5d9021338e37" containerName="revoke-keys"
Oct 30 01:23:05.808025 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.804961    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e427c9-5c89-4401-afa6-437676618dbf" containerName="istio-proxy"
Oct 30 01:23:05.808025 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.804975    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e427c9-5c89-4401-afa6-437676618dbf" containerName="istio-proxy"
Oct 30 01:23:05.813241 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.812870    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh"
Oct 30 01:23:05.828250 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.828225    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh"]
Oct 30 01:23:05.835595 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.835569    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:23:05.835744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.835592    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:23:05.931073 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.931027    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvbtx\" (UniqueName: \"kubernetes.io/projected/9204283f-afa2-46de-a0c4-2fa88960ac72-kube-api-access-nvbtx\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh\" (UID: \"9204283f-afa2-46de-a0c4-2fa88960ac72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh"
Oct 30 01:23:05.931235 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.931203    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9204283f-afa2-46de-a0c4-2fa88960ac72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh\" (UID: \"9204283f-afa2-46de-a0c4-2fa88960ac72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh"
Oct 30 01:23:05.931328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:05.931303    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9204283f-afa2-46de-a0c4-2fa88960ac72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh\" (UID: \"9204283f-afa2-46de-a0c4-2fa88960ac72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh"
Oct 30 01:23:06.032364 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:06.032333    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9204283f-afa2-46de-a0c4-2fa88960ac72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh\" (UID: \"9204283f-afa2-46de-a0c4-2fa88960ac72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh"
Oct 30 01:23:06.032558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:06.032438    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbtx\" (UniqueName: \"kubernetes.io/projected/9204283f-afa2-46de-a0c4-2fa88960ac72-kube-api-access-nvbtx\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh\" (UID: \"9204283f-afa2-46de-a0c4-2fa88960ac72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh"
Oct 30 01:23:06.032558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:06.032528    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9204283f-afa2-46de-a0c4-2fa88960ac72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh\" (UID: \"9204283f-afa2-46de-a0c4-2fa88960ac72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh"
Oct 30 01:23:06.033106 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:06.033079    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9204283f-afa2-46de-a0c4-2fa88960ac72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh\" (UID: \"9204283f-afa2-46de-a0c4-2fa88960ac72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh"
Oct 30 01:23:06.034845 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:06.034824    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9204283f-afa2-46de-a0c4-2fa88960ac72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh\" (UID: \"9204283f-afa2-46de-a0c4-2fa88960ac72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh"
Oct 30 01:23:06.063409 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:06.063353    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvbtx\" (UniqueName: \"kubernetes.io/projected/9204283f-afa2-46de-a0c4-2fa88960ac72-kube-api-access-nvbtx\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh\" (UID: \"9204283f-afa2-46de-a0c4-2fa88960ac72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh"
Oct 30 01:23:06.133198 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:06.133171    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh"
Oct 30 01:23:06.437683 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:23:06.437642    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9204283f_afa2_46de_a0c4_2fa88960ac72.slice/crio-83c277a238bcdf791a178a6b195870efa5d99b256a5b387dd051aa472e0b06b0 WatchSource:0}: Error finding container 83c277a238bcdf791a178a6b195870efa5d99b256a5b387dd051aa472e0b06b0: Status 404 returned error can't find the container with id 83c277a238bcdf791a178a6b195870efa5d99b256a5b387dd051aa472e0b06b0
Oct 30 01:23:06.438188 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:06.438162    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh"]
Oct 30 01:23:07.210458 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:07.210425    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh" event={"ID":"9204283f-afa2-46de-a0c4-2fa88960ac72","Type":"ContainerStarted","Data":"a25added2664f82202c6b91648014662deaaf45644976fa9e484cfa3b70ef2b3"}
Oct 30 01:23:07.210644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:07.210476    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh" event={"ID":"9204283f-afa2-46de-a0c4-2fa88960ac72","Type":"ContainerStarted","Data":"83c277a238bcdf791a178a6b195870efa5d99b256a5b387dd051aa472e0b06b0"}
Oct 30 01:23:07.274585 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:07.273445    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh" podStartSLOduration=2.273426429 podStartE2EDuration="2.273426429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:07.249912871 +0000 UTC m=+2881.106820627" watchObservedRunningTime="2026-10-30 01:23:07.273426429 +0000 UTC m=+2881.130334186"
Oct 30 01:23:08.223287 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:08.223251    6099 generic.go:356] "Generic (PLEG): container finished" podID="9204283f-afa2-46de-a0c4-2fa88960ac72" containerID="a25added2664f82202c6b91648014662deaaf45644976fa9e484cfa3b70ef2b3" exitCode=0
Oct 30 01:23:08.223723 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:08.223332    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh" event={"ID":"9204283f-afa2-46de-a0c4-2fa88960ac72","Type":"ContainerDied","Data":"a25added2664f82202c6b91648014662deaaf45644976fa9e484cfa3b70ef2b3"}
Oct 30 01:23:08.493971 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:08.493889    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq" podUID="6adc9cb6-daf1-4c37-b49f-96cfaf6a8387" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:08.494135 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:08.494019    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9" podUID="6b90542e-141d-47e4-8270-4aa8e67632ca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:09.258546 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:09.258510    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq"]
Oct 30 01:23:09.268243 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:09.268212    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2b4105-mmdrq"]
Oct 30 01:23:09.325426 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:09.325389    6099 patch_prober.go:29] interesting pod/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:23:09.325569 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:09.325450    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x" podUID="ed244c34-4856-46d8-a180-030fd7d4362e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:23:09.428950 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:09.428921    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh"
Oct 30 01:23:09.585613 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:09.585582    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9204283f-afa2-46de-a0c4-2fa88960ac72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9204283f-afa2-46de-a0c4-2fa88960ac72-maas-api-xks-ca\") pod \"9204283f-afa2-46de-a0c4-2fa88960ac72\" (UID: \"9204283f-afa2-46de-a0c4-2fa88960ac72\") "
Oct 30 01:23:09.585825 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:09.585755    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9204283f-afa2-46de-a0c4-2fa88960ac72-kube-api-access-nvbtx\" (UniqueName: \"kubernetes.io/projected/9204283f-afa2-46de-a0c4-2fa88960ac72-kube-api-access-nvbtx\") pod \"9204283f-afa2-46de-a0c4-2fa88960ac72\" (UID: \"9204283f-afa2-46de-a0c4-2fa88960ac72\") "
Oct 30 01:23:09.585825 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:09.585809    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9204283f-afa2-46de-a0c4-2fa88960ac72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9204283f-afa2-46de-a0c4-2fa88960ac72-maas-api-openshift-ca\") pod \"9204283f-afa2-46de-a0c4-2fa88960ac72\" (UID: \"9204283f-afa2-46de-a0c4-2fa88960ac72\") "
Oct 30 01:23:09.586156 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:09.586128    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9204283f-afa2-46de-a0c4-2fa88960ac72-maas-api-openshift-ca" pod "9204283f-afa2-46de-a0c4-2fa88960ac72" (UID: "9204283f-afa2-46de-a0c4-2fa88960ac72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:09.586473 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:09.586451    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9204283f-afa2-46de-a0c4-2fa88960ac72-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:09.587835 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:09.587811    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9204283f-afa2-46de-a0c4-2fa88960ac72-kube-api-access-nvbtx" pod "9204283f-afa2-46de-a0c4-2fa88960ac72" (UID: "9204283f-afa2-46de-a0c4-2fa88960ac72"). InnerVolumeSpecName "kube-api-access-nvbtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:09.587932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:09.587845    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9204283f-afa2-46de-a0c4-2fa88960ac72-maas-api-xks-ca" pod "9204283f-afa2-46de-a0c4-2fa88960ac72" (UID: "9204283f-afa2-46de-a0c4-2fa88960ac72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:09.688102 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:09.688067    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9204283f-afa2-46de-a0c4-2fa88960ac72-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:09.688102 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:09.688098    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvbtx\" (UniqueName: \"kubernetes.io/projected/9204283f-afa2-46de-a0c4-2fa88960ac72-kube-api-access-nvbtx\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:10.250784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.250757    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh"
Oct 30 01:23:10.251008 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.250759    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh" event={"ID":"9204283f-afa2-46de-a0c4-2fa88960ac72","Type":"ContainerDied","Data":"83c277a238bcdf791a178a6b195870efa5d99b256a5b387dd051aa472e0b06b0"}
Oct 30 01:23:10.251008 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.250885    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83c277a238bcdf791a178a6b195870efa5d99b256a5b387dd051aa472e0b06b0"
Oct 30 01:23:10.263662 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.263639    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305f33b2-e183-44b6-9f89-e90047f939e8" path="/var/lib/kubelet/pods/305f33b2-e183-44b6-9f89-e90047f939e8/volumes"
Oct 30 01:23:10.538523 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.538499    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:23:10.702851 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.702816    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-workload-certs\") pod \"ed244c34-4856-46d8-a180-030fd7d4362e\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") "
Oct 30 01:23:10.702851 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.702850    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-workload-socket\") pod \"ed244c34-4856-46d8-a180-030fd7d4362e\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") "
Oct 30 01:23:10.703059 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.702990    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-istio-data\") pod \"ed244c34-4856-46d8-a180-030fd7d4362e\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") "
Oct 30 01:23:10.703059 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.703022    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-istio-envoy\") pod \"ed244c34-4856-46d8-a180-030fd7d4362e\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") "
Oct 30 01:23:10.703194 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.703102    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed244c34-4856-46d8-a180-030fd7d4362e-kube-api-access-jqjzr\" (UniqueName: \"kubernetes.io/projected/ed244c34-4856-46d8-a180-030fd7d4362e-kube-api-access-jqjzr\") pod \"ed244c34-4856-46d8-a180-030fd7d4362e\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") "
Oct 30 01:23:10.703244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.703100    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-workload-certs" pod "ed244c34-4856-46d8-a180-030fd7d4362e" (UID: "ed244c34-4856-46d8-a180-030fd7d4362e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:10.703244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.703196    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed244c34-4856-46d8-a180-030fd7d4362e-istio-token\" (UniqueName: \"kubernetes.io/projected/ed244c34-4856-46d8-a180-030fd7d4362e-istio-token\") pod \"ed244c34-4856-46d8-a180-030fd7d4362e\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") "
Oct 30 01:23:10.703244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.703181    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-workload-socket" pod "ed244c34-4856-46d8-a180-030fd7d4362e" (UID: "ed244c34-4856-46d8-a180-030fd7d4362e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:10.703244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.703234    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed244c34-4856-46d8-a180-030fd7d4362e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed244c34-4856-46d8-a180-030fd7d4362e-istio-podinfo\") pod \"ed244c34-4856-46d8-a180-030fd7d4362e\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") "
Oct 30 01:23:10.703461 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.703270    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-istio-data" pod "ed244c34-4856-46d8-a180-030fd7d4362e" (UID: "ed244c34-4856-46d8-a180-030fd7d4362e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:10.703461 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.703274    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-credential-socket\") pod \"ed244c34-4856-46d8-a180-030fd7d4362e\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") "
Oct 30 01:23:10.703461 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.703376    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed244c34-4856-46d8-a180-030fd7d4362e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed244c34-4856-46d8-a180-030fd7d4362e-istiod-ca-cert\") pod \"ed244c34-4856-46d8-a180-030fd7d4362e\" (UID: \"ed244c34-4856-46d8-a180-030fd7d4362e\") "
Oct 30 01:23:10.703629 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.703487    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-credential-socket" pod "ed244c34-4856-46d8-a180-030fd7d4362e" (UID: "ed244c34-4856-46d8-a180-030fd7d4362e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:10.703763 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.703739    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed244c34-4856-46d8-a180-030fd7d4362e-istiod-ca-cert" pod "ed244c34-4856-46d8-a180-030fd7d4362e" (UID: "ed244c34-4856-46d8-a180-030fd7d4362e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:10.704152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.704129    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:10.704210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.704160    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:10.704210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.704176    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed244c34-4856-46d8-a180-030fd7d4362e-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:10.704210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.704189    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:10.704210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.704203    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:10.705448 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.705423    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed244c34-4856-46d8-a180-030fd7d4362e-istio-podinfo" pod "ed244c34-4856-46d8-a180-030fd7d4362e" (UID: "ed244c34-4856-46d8-a180-030fd7d4362e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:10.705448 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.705437    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed244c34-4856-46d8-a180-030fd7d4362e-istio-token" pod "ed244c34-4856-46d8-a180-030fd7d4362e" (UID: "ed244c34-4856-46d8-a180-030fd7d4362e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:10.705742 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.705696    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-istio-envoy" pod "ed244c34-4856-46d8-a180-030fd7d4362e" (UID: "ed244c34-4856-46d8-a180-030fd7d4362e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:10.705836 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.705742    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed244c34-4856-46d8-a180-030fd7d4362e-kube-api-access-jqjzr" pod "ed244c34-4856-46d8-a180-030fd7d4362e" (UID: "ed244c34-4856-46d8-a180-030fd7d4362e"). InnerVolumeSpecName "kube-api-access-jqjzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:10.805949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.805919    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed244c34-4856-46d8-a180-030fd7d4362e-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:10.805949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.805948    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqjzr\" (UniqueName: \"kubernetes.io/projected/ed244c34-4856-46d8-a180-030fd7d4362e-kube-api-access-jqjzr\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:10.806152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.805958    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed244c34-4856-46d8-a180-030fd7d4362e-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:10.806152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:10.805967    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed244c34-4856-46d8-a180-030fd7d4362e-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:11.259859 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:11.259783    6099 generic.go:356] "Generic (PLEG): container finished" podID="ed244c34-4856-46d8-a180-030fd7d4362e" containerID="fd049efca35e876e1ce093c359eeb2f77c28ba75ab448e497f9b96c9b7dcc672" exitCode=0
Oct 30 01:23:11.259859 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:11.259821    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"
Oct 30 01:23:11.260048 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:11.259862    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x" event={"ID":"ed244c34-4856-46d8-a180-030fd7d4362e","Type":"ContainerDied","Data":"fd049efca35e876e1ce093c359eeb2f77c28ba75ab448e497f9b96c9b7dcc672"}
Oct 30 01:23:11.260048 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:11.259914    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x" event={"ID":"ed244c34-4856-46d8-a180-030fd7d4362e","Type":"ContainerDied","Data":"eef73b7a14183d74906efbb7166c5ed1ab01bb2fa8ff7f012172507f5d9a312d"}
Oct 30 01:23:11.260048 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:11.259942    6099 scope.go:122] "RemoveContainer" containerID="fd049efca35e876e1ce093c359eeb2f77c28ba75ab448e497f9b96c9b7dcc672"
Oct 30 01:23:11.271739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:11.271694    6099 scope.go:122] "RemoveContainer" containerID="fd049efca35e876e1ce093c359eeb2f77c28ba75ab448e497f9b96c9b7dcc672"
Oct 30 01:23:11.272058 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:23:11.271967    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd049efca35e876e1ce093c359eeb2f77c28ba75ab448e497f9b96c9b7dcc672\": container with ID starting with fd049efca35e876e1ce093c359eeb2f77c28ba75ab448e497f9b96c9b7dcc672 not found: ID does not exist" containerID="fd049efca35e876e1ce093c359eeb2f77c28ba75ab448e497f9b96c9b7dcc672"
Oct 30 01:23:11.272058 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:11.271992    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd049efca35e876e1ce093c359eeb2f77c28ba75ab448e497f9b96c9b7dcc672"} err="failed to get container status \"fd049efca35e876e1ce093c359eeb2f77c28ba75ab448e497f9b96c9b7dcc672\": rpc error: code = NotFound desc = could not find container \"fd049efca35e876e1ce093c359eeb2f77c28ba75ab448e497f9b96c9b7dcc672\": container with ID starting with fd049efca35e876e1ce093c359eeb2f77c28ba75ab448e497f9b96c9b7dcc672 not found: ID does not exist"
Oct 30 01:23:11.325756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:11.325694    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"]
Oct 30 01:23:11.336624 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:11.336592    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-969b92-openshift-default-b496c6b6f-ftc6x"]
Oct 30 01:23:12.263466 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:12.263431    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed244c34-4856-46d8-a180-030fd7d4362e" path="/var/lib/kubelet/pods/ed244c34-4856-46d8-a180-030fd7d4362e/volumes"
Oct 30 01:23:15.103597 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:15.103568    6099 scope.go:122] "RemoveContainer" containerID="030ee7d7bc1dfd3d6150d889b785b249161e12d8010c6903639a51550b636745"
Oct 30 01:23:16.323549 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:16.323510    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"]
Oct 30 01:23:16.324327 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:16.323776    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh" podUID="58d5ec0f-0a36-486c-b391-cc0660afa52f" containerName="istio-proxy" containerID="cri-o://111f83b50a39d29877b14484e33ad579901c11b3bf2cdcfd3e7428db1d3c89bd" gracePeriod=30
Oct 30 01:23:16.631601 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:23:16.631515    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:23:16.661182 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:16.661154    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4"]
Oct 30 01:23:17.356909 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:17.356876    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq"]
Oct 30 01:23:18.493159 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:18.493081    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9" podUID="6b90542e-141d-47e4-8270-4aa8e67632ca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:18.493588 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:18.493152    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq" podUID="6adc9cb6-daf1-4c37-b49f-96cfaf6a8387" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:18.929670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:18.929627    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8"]
Oct 30 01:23:18.942371 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:18.942335    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6add35-7ccw8"]
Oct 30 01:23:20.261667 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:20.261635    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af17237-0f9e-435c-b453-ced511ca7425" path="/var/lib/kubelet/pods/9af17237-0f9e-435c-b453-ced511ca7425/volumes"
Oct 30 01:23:21.657686 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.657657    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 01:23:21.662484 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.662446    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4" podUID="293c28e2-6799-412b-9373-1e418269cd10" containerName="payload-pre-processing" containerID="cri-o://813f5334bfbabc156b72150f61202f621b6a85a7b2dedab9119f67eec7415de8" gracePeriod=25
Oct 30 01:23:21.698261 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.698231    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq"
Oct 30 01:23:21.771643 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.771618    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9"
Oct 30 01:23:21.772089 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.772061    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-workload-certs\") pod \"58d5ec0f-0a36-486c-b391-cc0660afa52f\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") "
Oct 30 01:23:21.772203 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.772130    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-credential-socket\") pod \"58d5ec0f-0a36-486c-b391-cc0660afa52f\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") "
Oct 30 01:23:21.772203 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.772168    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-podinfo\") pod \"58d5ec0f-0a36-486c-b391-cc0660afa52f\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") "
Oct 30 01:23:21.772326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.772240    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-workload-socket\") pod \"58d5ec0f-0a36-486c-b391-cc0660afa52f\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") "
Oct 30 01:23:21.772326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.772290    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-envoy\") pod \"58d5ec0f-0a36-486c-b391-cc0660afa52f\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") "
Oct 30 01:23:21.772444 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.772377    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d5ec0f-0a36-486c-b391-cc0660afa52f-kube-api-access-9w6h5\" (UniqueName: \"kubernetes.io/projected/58d5ec0f-0a36-486c-b391-cc0660afa52f-kube-api-access-9w6h5\") pod \"58d5ec0f-0a36-486c-b391-cc0660afa52f\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") "
Oct 30 01:23:21.772444 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.772374    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-workload-certs" pod "58d5ec0f-0a36-486c-b391-cc0660afa52f" (UID: "58d5ec0f-0a36-486c-b391-cc0660afa52f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:21.772444 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.772438    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-data\") pod \"58d5ec0f-0a36-486c-b391-cc0660afa52f\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") "
Oct 30 01:23:21.772676 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.772475    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6adc9cb6-daf1-4c37-b49f-96cfaf6a8387-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6adc9cb6-daf1-4c37-b49f-96cfaf6a8387-plugins-config-volume\") pod \"6adc9cb6-daf1-4c37-b49f-96cfaf6a8387\" (UID: \"6adc9cb6-daf1-4c37-b49f-96cfaf6a8387\") "
Oct 30 01:23:21.772676 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.772519    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-token\" (UniqueName: \"kubernetes.io/projected/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-token\") pod \"58d5ec0f-0a36-486c-b391-cc0660afa52f\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") "
Oct 30 01:23:21.772676 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.772568    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58d5ec0f-0a36-486c-b391-cc0660afa52f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d5ec0f-0a36-486c-b391-cc0660afa52f-istiod-ca-cert\") pod \"58d5ec0f-0a36-486c-b391-cc0660afa52f\" (UID: \"58d5ec0f-0a36-486c-b391-cc0660afa52f\") "
Oct 30 01:23:21.772676 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.772603    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6adc9cb6-daf1-4c37-b49f-96cfaf6a8387-kube-api-access-lx8h8\" (UniqueName: \"kubernetes.io/projected/6adc9cb6-daf1-4c37-b49f-96cfaf6a8387-kube-api-access-lx8h8\") pod \"6adc9cb6-daf1-4c37-b49f-96cfaf6a8387\" (UID: \"6adc9cb6-daf1-4c37-b49f-96cfaf6a8387\") "
Oct 30 01:23:21.772915 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.772839    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-credential-socket" pod "58d5ec0f-0a36-486c-b391-cc0660afa52f" (UID: "58d5ec0f-0a36-486c-b391-cc0660afa52f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:21.772983 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.772970    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-data" pod "58d5ec0f-0a36-486c-b391-cc0660afa52f" (UID: "58d5ec0f-0a36-486c-b391-cc0660afa52f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:21.773189 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.773176    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:21.773230 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.773193    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:21.773230 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.773202    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:21.773362 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.773349    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-workload-socket" pod "58d5ec0f-0a36-486c-b391-cc0660afa52f" (UID: "58d5ec0f-0a36-486c-b391-cc0660afa52f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:21.773458 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.773433    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6adc9cb6-daf1-4c37-b49f-96cfaf6a8387-plugins-config-volume" pod "6adc9cb6-daf1-4c37-b49f-96cfaf6a8387" (UID: "6adc9cb6-daf1-4c37-b49f-96cfaf6a8387"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:21.774382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.774351    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d5ec0f-0a36-486c-b391-cc0660afa52f-istiod-ca-cert" pod "58d5ec0f-0a36-486c-b391-cc0660afa52f" (UID: "58d5ec0f-0a36-486c-b391-cc0660afa52f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:21.774840 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.774818    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-podinfo" pod "58d5ec0f-0a36-486c-b391-cc0660afa52f" (UID: "58d5ec0f-0a36-486c-b391-cc0660afa52f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:21.775510 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.775484    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-envoy" pod "58d5ec0f-0a36-486c-b391-cc0660afa52f" (UID: "58d5ec0f-0a36-486c-b391-cc0660afa52f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:21.775791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.775767    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-token" pod "58d5ec0f-0a36-486c-b391-cc0660afa52f" (UID: "58d5ec0f-0a36-486c-b391-cc0660afa52f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:21.776119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.776095    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6adc9cb6-daf1-4c37-b49f-96cfaf6a8387-kube-api-access-lx8h8" pod "6adc9cb6-daf1-4c37-b49f-96cfaf6a8387" (UID: "6adc9cb6-daf1-4c37-b49f-96cfaf6a8387"). InnerVolumeSpecName "kube-api-access-lx8h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:21.776220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.776201    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d5ec0f-0a36-486c-b391-cc0660afa52f-kube-api-access-9w6h5" pod "58d5ec0f-0a36-486c-b391-cc0660afa52f" (UID: "58d5ec0f-0a36-486c-b391-cc0660afa52f"). InnerVolumeSpecName "kube-api-access-9w6h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:21.874266 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.874187    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b90542e-141d-47e4-8270-4aa8e67632ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b90542e-141d-47e4-8270-4aa8e67632ca-plugins-config-volume\") pod \"6b90542e-141d-47e4-8270-4aa8e67632ca\" (UID: \"6b90542e-141d-47e4-8270-4aa8e67632ca\") "
Oct 30 01:23:21.874408 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.874289    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b90542e-141d-47e4-8270-4aa8e67632ca-kube-api-access-4jppb\" (UniqueName: \"kubernetes.io/projected/6b90542e-141d-47e4-8270-4aa8e67632ca-kube-api-access-4jppb\") pod \"6b90542e-141d-47e4-8270-4aa8e67632ca\" (UID: \"6b90542e-141d-47e4-8270-4aa8e67632ca\") "
Oct 30 01:23:21.874612 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.874589    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b90542e-141d-47e4-8270-4aa8e67632ca-plugins-config-volume" pod "6b90542e-141d-47e4-8270-4aa8e67632ca" (UID: "6b90542e-141d-47e4-8270-4aa8e67632ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:21.874892 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.874870    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b90542e-141d-47e4-8270-4aa8e67632ca-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:21.874892 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.874891    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:21.875041 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.874901    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:21.875041 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.874914    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w6h5\" (UniqueName: \"kubernetes.io/projected/58d5ec0f-0a36-486c-b391-cc0660afa52f-kube-api-access-9w6h5\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:21.875041 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.874926    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6adc9cb6-daf1-4c37-b49f-96cfaf6a8387-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:21.875041 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.874938    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:21.875041 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.874945    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d5ec0f-0a36-486c-b391-cc0660afa52f-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:21.875041 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.874953    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx8h8\" (UniqueName: \"kubernetes.io/projected/6adc9cb6-daf1-4c37-b49f-96cfaf6a8387-kube-api-access-lx8h8\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:21.875041 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.874961    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d5ec0f-0a36-486c-b391-cc0660afa52f-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:21.876268 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.876238    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b90542e-141d-47e4-8270-4aa8e67632ca-kube-api-access-4jppb" pod "6b90542e-141d-47e4-8270-4aa8e67632ca" (UID: "6b90542e-141d-47e4-8270-4aa8e67632ca"). InnerVolumeSpecName "kube-api-access-4jppb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:21.975968 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:21.975938    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jppb\" (UniqueName: \"kubernetes.io/projected/6b90542e-141d-47e4-8270-4aa8e67632ca-kube-api-access-4jppb\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:22.358122 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.358035    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq" podUID="e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55" containerName="payload-processing" containerID="cri-o://01e4eac101cbaddbfb9bf52ab5a1ed2ecbae6d080e7101922f54a7aaf06c27b3" gracePeriod=25
Oct 30 01:23:22.369431 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.369403    6099 generic.go:356] "Generic (PLEG): container finished" podID="58d5ec0f-0a36-486c-b391-cc0660afa52f" containerID="111f83b50a39d29877b14484e33ad579901c11b3bf2cdcfd3e7428db1d3c89bd" exitCode=0
Oct 30 01:23:22.369570 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.369463    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"
Oct 30 01:23:22.369570 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.369487    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh" event={"ID":"58d5ec0f-0a36-486c-b391-cc0660afa52f","Type":"ContainerDied","Data":"111f83b50a39d29877b14484e33ad579901c11b3bf2cdcfd3e7428db1d3c89bd"}
Oct 30 01:23:22.369570 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.369547    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh" event={"ID":"58d5ec0f-0a36-486c-b391-cc0660afa52f","Type":"ContainerDied","Data":"ec8f53a910db4b4c32b5313db416065eaade8ebeb6cb49022cbf1fbc29852157"}
Oct 30 01:23:22.369734 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.369582    6099 scope.go:122] "RemoveContainer" containerID="111f83b50a39d29877b14484e33ad579901c11b3bf2cdcfd3e7428db1d3c89bd"
Oct 30 01:23:22.370991 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.370972    6099 generic.go:356] "Generic (PLEG): container finished" podID="6b90542e-141d-47e4-8270-4aa8e67632ca" containerID="a2fc4d297acb093a1d6c25691d94c54589074f387be1260ecbf063c85ec8f88d" exitCode=0
Oct 30 01:23:22.371095 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.371022    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9"
Oct 30 01:23:22.371095 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.371057    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9" event={"ID":"6b90542e-141d-47e4-8270-4aa8e67632ca","Type":"ContainerDied","Data":"a2fc4d297acb093a1d6c25691d94c54589074f387be1260ecbf063c85ec8f88d"}
Oct 30 01:23:22.371228 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.371104    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9" event={"ID":"6b90542e-141d-47e4-8270-4aa8e67632ca","Type":"ContainerDied","Data":"a41f1cf6ccb6afa41bb9bf58ea75ab2b413f058608ca512c7848596e80dfe1ba"}
Oct 30 01:23:22.372525 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.372502    6099 generic.go:356] "Generic (PLEG): container finished" podID="6adc9cb6-daf1-4c37-b49f-96cfaf6a8387" containerID="29e1ae1ddaecfe7808b1edbab9ed6f73d8303ecb59bec59eff0839d46ada3a6d" exitCode=0
Oct 30 01:23:22.372626 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.372536    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq"
Oct 30 01:23:22.372626 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.372587    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq" event={"ID":"6adc9cb6-daf1-4c37-b49f-96cfaf6a8387","Type":"ContainerDied","Data":"29e1ae1ddaecfe7808b1edbab9ed6f73d8303ecb59bec59eff0839d46ada3a6d"}
Oct 30 01:23:22.372719 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.372627    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq" event={"ID":"6adc9cb6-daf1-4c37-b49f-96cfaf6a8387","Type":"ContainerDied","Data":"9612fd02cbe7c0ce581456b03aaec4d515dbfbc8bc3e7e728e4ddc3b006e7eb5"}
Oct 30 01:23:22.471499 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.471473    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9"]
Oct 30 01:23:22.473254 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.473131    6099 scope.go:122] "RemoveContainer" containerID="111f83b50a39d29877b14484e33ad579901c11b3bf2cdcfd3e7428db1d3c89bd"
Oct 30 01:23:22.473558 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:23:22.473529    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111f83b50a39d29877b14484e33ad579901c11b3bf2cdcfd3e7428db1d3c89bd\": container with ID starting with 111f83b50a39d29877b14484e33ad579901c11b3bf2cdcfd3e7428db1d3c89bd not found: ID does not exist" containerID="111f83b50a39d29877b14484e33ad579901c11b3bf2cdcfd3e7428db1d3c89bd"
Oct 30 01:23:22.473633 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.473567    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111f83b50a39d29877b14484e33ad579901c11b3bf2cdcfd3e7428db1d3c89bd"} err="failed to get container status \"111f83b50a39d29877b14484e33ad579901c11b3bf2cdcfd3e7428db1d3c89bd\": rpc error: code = NotFound desc = could not find container \"111f83b50a39d29877b14484e33ad579901c11b3bf2cdcfd3e7428db1d3c89bd\": container with ID starting with 111f83b50a39d29877b14484e33ad579901c11b3bf2cdcfd3e7428db1d3c89bd not found: ID does not exist"
Oct 30 01:23:22.473633 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.473587    6099 scope.go:122] "RemoveContainer" containerID="a2fc4d297acb093a1d6c25691d94c54589074f387be1260ecbf063c85ec8f88d"
Oct 30 01:23:22.479435 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.479415    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-969b92-57ff684f47-645k9"]
Oct 30 01:23:22.484209 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.484190    6099 scope.go:122] "RemoveContainer" containerID="a2fc4d297acb093a1d6c25691d94c54589074f387be1260ecbf063c85ec8f88d"
Oct 30 01:23:22.484466 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:23:22.484447    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2fc4d297acb093a1d6c25691d94c54589074f387be1260ecbf063c85ec8f88d\": container with ID starting with a2fc4d297acb093a1d6c25691d94c54589074f387be1260ecbf063c85ec8f88d not found: ID does not exist" containerID="a2fc4d297acb093a1d6c25691d94c54589074f387be1260ecbf063c85ec8f88d"
Oct 30 01:23:22.484534 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.484474    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2fc4d297acb093a1d6c25691d94c54589074f387be1260ecbf063c85ec8f88d"} err="failed to get container status \"a2fc4d297acb093a1d6c25691d94c54589074f387be1260ecbf063c85ec8f88d\": rpc error: code = NotFound desc = could not find container \"a2fc4d297acb093a1d6c25691d94c54589074f387be1260ecbf063c85ec8f88d\": container with ID starting with a2fc4d297acb093a1d6c25691d94c54589074f387be1260ecbf063c85ec8f88d not found: ID does not exist"
Oct 30 01:23:22.484534 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.484493    6099 scope.go:122] "RemoveContainer" containerID="29e1ae1ddaecfe7808b1edbab9ed6f73d8303ecb59bec59eff0839d46ada3a6d"
Oct 30 01:23:22.494867 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.494846    6099 scope.go:122] "RemoveContainer" containerID="29e1ae1ddaecfe7808b1edbab9ed6f73d8303ecb59bec59eff0839d46ada3a6d"
Oct 30 01:23:22.495249 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:23:22.495207    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e1ae1ddaecfe7808b1edbab9ed6f73d8303ecb59bec59eff0839d46ada3a6d\": container with ID starting with 29e1ae1ddaecfe7808b1edbab9ed6f73d8303ecb59bec59eff0839d46ada3a6d not found: ID does not exist" containerID="29e1ae1ddaecfe7808b1edbab9ed6f73d8303ecb59bec59eff0839d46ada3a6d"
Oct 30 01:23:22.495388 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.495258    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e1ae1ddaecfe7808b1edbab9ed6f73d8303ecb59bec59eff0839d46ada3a6d"} err="failed to get container status \"29e1ae1ddaecfe7808b1edbab9ed6f73d8303ecb59bec59eff0839d46ada3a6d\": rpc error: code = NotFound desc = could not find container \"29e1ae1ddaecfe7808b1edbab9ed6f73d8303ecb59bec59eff0839d46ada3a6d\": container with ID starting with 29e1ae1ddaecfe7808b1edbab9ed6f73d8303ecb59bec59eff0839d46ada3a6d not found: ID does not exist"
Oct 30 01:23:22.517976 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.514855    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"]
Oct 30 01:23:22.519018 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.518992    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2b4105-openshift-default-d944d864f-9wxjh"]
Oct 30 01:23:22.617383 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.617343    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq"]
Oct 30 01:23:22.621584 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.621555    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-969b92-75f748cc6b-bq5bq"]
Oct 30 01:23:22.661581 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.661559    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq"
Oct 30 01:23:22.808420 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.808334    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55-plugins-config-volume\") pod \"e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55\" (UID: \"e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55\") "
Oct 30 01:23:22.808420 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.808404    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55-kube-api-access-zbxlg\" (UniqueName: \"kubernetes.io/projected/e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55-kube-api-access-zbxlg\") pod \"e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55\" (UID: \"e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55\") "
Oct 30 01:23:22.808813 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.808785    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55-plugins-config-volume" pod "e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55" (UID: "e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:22.809095 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.809077    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:22.810531 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.810502    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55-kube-api-access-zbxlg" pod "e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55" (UID: "e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55"). InnerVolumeSpecName "kube-api-access-zbxlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:22.910278 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:22.910251    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbxlg\" (UniqueName: \"kubernetes.io/projected/e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55-kube-api-access-zbxlg\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.381354 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:23.381316    6099 generic.go:356] "Generic (PLEG): container finished" podID="e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55" containerID="01e4eac101cbaddbfb9bf52ab5a1ed2ecbae6d080e7101922f54a7aaf06c27b3" exitCode=0
Oct 30 01:23:23.381528 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:23.381359    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq"
Oct 30 01:23:23.381528 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:23.381402    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq" event={"ID":"e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55","Type":"ContainerDied","Data":"01e4eac101cbaddbfb9bf52ab5a1ed2ecbae6d080e7101922f54a7aaf06c27b3"}
Oct 30 01:23:23.381528 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:23.381454    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq" event={"ID":"e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55","Type":"ContainerDied","Data":"b4a66a625cb6a001c987e8f558b0def463da8c1bc74d543c7c936daf03b217f1"}
Oct 30 01:23:23.381528 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:23.381485    6099 scope.go:122] "RemoveContainer" containerID="01e4eac101cbaddbfb9bf52ab5a1ed2ecbae6d080e7101922f54a7aaf06c27b3"
Oct 30 01:23:23.394131 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:23.393917    6099 scope.go:122] "RemoveContainer" containerID="01e4eac101cbaddbfb9bf52ab5a1ed2ecbae6d080e7101922f54a7aaf06c27b3"
Oct 30 01:23:23.394438 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:23:23.394415    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01e4eac101cbaddbfb9bf52ab5a1ed2ecbae6d080e7101922f54a7aaf06c27b3\": container with ID starting with 01e4eac101cbaddbfb9bf52ab5a1ed2ecbae6d080e7101922f54a7aaf06c27b3 not found: ID does not exist" containerID="01e4eac101cbaddbfb9bf52ab5a1ed2ecbae6d080e7101922f54a7aaf06c27b3"
Oct 30 01:23:23.394544 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:23.394448    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01e4eac101cbaddbfb9bf52ab5a1ed2ecbae6d080e7101922f54a7aaf06c27b3"} err="failed to get container status \"01e4eac101cbaddbfb9bf52ab5a1ed2ecbae6d080e7101922f54a7aaf06c27b3\": rpc error: code = NotFound desc = could not find container \"01e4eac101cbaddbfb9bf52ab5a1ed2ecbae6d080e7101922f54a7aaf06c27b3\": container with ID starting with 01e4eac101cbaddbfb9bf52ab5a1ed2ecbae6d080e7101922f54a7aaf06c27b3 not found: ID does not exist"
Oct 30 01:23:23.434920 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:23.434889    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq"]
Oct 30 01:23:23.445583 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:23.445554    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e9921-5f86b9565b-d5wpq"]
Oct 30 01:23:24.282924 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.282890    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d5ec0f-0a36-486c-b391-cc0660afa52f" path="/var/lib/kubelet/pods/58d5ec0f-0a36-486c-b391-cc0660afa52f/volumes"
Oct 30 01:23:24.283456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.283376    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6adc9cb6-daf1-4c37-b49f-96cfaf6a8387" path="/var/lib/kubelet/pods/6adc9cb6-daf1-4c37-b49f-96cfaf6a8387/volumes"
Oct 30 01:23:24.283819 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.283801    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b90542e-141d-47e4-8270-4aa8e67632ca" path="/var/lib/kubelet/pods/6b90542e-141d-47e4-8270-4aa8e67632ca/volumes"
Oct 30 01:23:24.285931 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.285366    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55" path="/var/lib/kubelet/pods/e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55/volumes"
Oct 30 01:23:24.293245 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.293222    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f"]
Oct 30 01:23:24.293894 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.293876    6099 cpu_manager.go:414] "containerMap: removing container" podUID="9204283f-afa2-46de-a0c4-2fa88960ac72" containerName="revoke-keys"
Oct 30 01:23:24.293994 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.293896    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9204283f-afa2-46de-a0c4-2fa88960ac72" containerName="revoke-keys"
Oct 30 01:23:24.293994 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.293943    6099 cpu_manager.go:414] "containerMap: removing container" podUID="58d5ec0f-0a36-486c-b391-cc0660afa52f" containerName="istio-proxy"
Oct 30 01:23:24.293994 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.293952    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d5ec0f-0a36-486c-b391-cc0660afa52f" containerName="istio-proxy"
Oct 30 01:23:24.293994 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.293963    6099 cpu_manager.go:414] "containerMap: removing container" podUID="6b90542e-141d-47e4-8270-4aa8e67632ca" containerName="payload-pre-processing"
Oct 30 01:23:24.293994 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.293970    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b90542e-141d-47e4-8270-4aa8e67632ca" containerName="payload-pre-processing"
Oct 30 01:23:24.294260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.294005    6099 cpu_manager.go:414] "containerMap: removing container" podUID="ed244c34-4856-46d8-a180-030fd7d4362e" containerName="istio-proxy"
Oct 30 01:23:24.294260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.294012    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed244c34-4856-46d8-a180-030fd7d4362e" containerName="istio-proxy"
Oct 30 01:23:24.294260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.294026    6099 cpu_manager.go:414] "containerMap: removing container" podUID="6adc9cb6-daf1-4c37-b49f-96cfaf6a8387" containerName="payload-processing"
Oct 30 01:23:24.294260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.294033    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6adc9cb6-daf1-4c37-b49f-96cfaf6a8387" containerName="payload-processing"
Oct 30 01:23:24.294260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.294056    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55" containerName="payload-processing"
Oct 30 01:23:24.294260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.294063    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55" containerName="payload-processing"
Oct 30 01:23:24.294260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.294208    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d5ec0f-0a36-486c-b391-cc0660afa52f" containerName="istio-proxy"
Oct 30 01:23:24.294260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.294220    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b90542e-141d-47e4-8270-4aa8e67632ca" containerName="payload-pre-processing"
Oct 30 01:23:24.294260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.294236    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed244c34-4856-46d8-a180-030fd7d4362e" containerName="istio-proxy"
Oct 30 01:23:24.294260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.294247    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6adc9cb6-daf1-4c37-b49f-96cfaf6a8387" containerName="payload-processing"
Oct 30 01:23:24.294260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.294258    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ff88a6-ef5c-44d7-ab3c-93d3d2e8cd55" containerName="payload-processing"
Oct 30 01:23:24.294891 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.294271    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9204283f-afa2-46de-a0c4-2fa88960ac72" containerName="revoke-keys"
Oct 30 01:23:24.299617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.299596    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f"
Oct 30 01:23:24.313203 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.313179    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f"]
Oct 30 01:23:24.321553 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.321527    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:23:24.321673 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.321604    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:23:24.431339 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.431309    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a32a218-ktz5f\" (UID: \"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f"
Oct 30 01:23:24.431502 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.431355    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a32a218-ktz5f\" (UID: \"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f"
Oct 30 01:23:24.431558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.431499    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c64rv\" (UniqueName: \"kubernetes.io/projected/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-kube-api-access-c64rv\") pod \"maas-api-revoke-sub-6a32a218-ktz5f\" (UID: \"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f"
Oct 30 01:23:24.533005 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.532916    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c64rv\" (UniqueName: \"kubernetes.io/projected/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-kube-api-access-c64rv\") pod \"maas-api-revoke-sub-6a32a218-ktz5f\" (UID: \"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f"
Oct 30 01:23:24.533152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.533109    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a32a218-ktz5f\" (UID: \"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f"
Oct 30 01:23:24.533225 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.533152    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a32a218-ktz5f\" (UID: \"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f"
Oct 30 01:23:24.533791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.533763    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a32a218-ktz5f\" (UID: \"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f"
Oct 30 01:23:24.535851 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.535831    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a32a218-ktz5f\" (UID: \"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f"
Oct 30 01:23:24.555951 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.555923    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c64rv\" (UniqueName: \"kubernetes.io/projected/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-kube-api-access-c64rv\") pod \"maas-api-revoke-sub-6a32a218-ktz5f\" (UID: \"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f"
Oct 30 01:23:24.570252 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.570226    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b"]
Oct 30 01:23:24.575030 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.575007    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ea2547-x2n2b"]
Oct 30 01:23:24.614967 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.614943    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f"
Oct 30 01:23:24.933245 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:24.933216    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f"]
Oct 30 01:23:24.934035 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:23:24.934003    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf5c354_53f3_4169_8c2b_42ea97f1b2a6.slice/crio-6df0ca14f8f8195b0c5a661872c49608c0b6d454cc2dc827e28e0ac8e79a073e WatchSource:0}: Error finding container 6df0ca14f8f8195b0c5a661872c49608c0b6d454cc2dc827e28e0ac8e79a073e: Status 404 returned error can't find the container with id 6df0ca14f8f8195b0c5a661872c49608c0b6d454cc2dc827e28e0ac8e79a073e
Oct 30 01:23:25.406264 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:25.406223    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f" event={"ID":"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6","Type":"ContainerStarted","Data":"ad46b6823071dcc5745541ed04bae8afdf0e810025d418d201b66d38ffe7bc45"}
Oct 30 01:23:25.406632 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:25.406287    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f" event={"ID":"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6","Type":"ContainerStarted","Data":"6df0ca14f8f8195b0c5a661872c49608c0b6d454cc2dc827e28e0ac8e79a073e"}
Oct 30 01:23:25.448842 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:25.448760    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f" podStartSLOduration=1.4487462039999999 podStartE2EDuration="1.448746204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:25.445668145 +0000 UTC m=+2899.302575902" watchObservedRunningTime="2026-10-30 01:23:25.448746204 +0000 UTC m=+2899.305653958"
Oct 30 01:23:26.199974 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:26.199892    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"]
Oct 30 01:23:26.200151 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:26.200124    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp" podUID="e5edebd8-443c-42d5-afc8-2c5495b4af01" containerName="istio-proxy" containerID="cri-o://96c5b021e92d1ef9159f2cad45b6085bbd0fe0bc3277f456b8f35733b29fb1d6" gracePeriod=30
Oct 30 01:23:26.262662 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:26.262600    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2c329c9-52a2-47e9-9ba0-5d9021338e37" path="/var/lib/kubelet/pods/a2c329c9-52a2-47e9-9ba0-5d9021338e37/volumes"
Oct 30 01:23:26.414443 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:26.414405    6099 generic.go:356] "Generic (PLEG): container finished" podID="4cf5c354-53f3-4169-8c2b-42ea97f1b2a6" containerID="ad46b6823071dcc5745541ed04bae8afdf0e810025d418d201b66d38ffe7bc45" exitCode=0
Oct 30 01:23:26.414904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:26.414481    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f" event={"ID":"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6","Type":"ContainerDied","Data":"ad46b6823071dcc5745541ed04bae8afdf0e810025d418d201b66d38ffe7bc45"}
Oct 30 01:23:26.852217 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:26.852179    6099 patch_prober.go:29] interesting pod/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:23:26.852409 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:26.852240    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp" podUID="e5edebd8-443c-42d5-afc8-2c5495b4af01" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:23:27.588178 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:27.588150    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f"
Oct 30 01:23:27.675607 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:27.675538    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-kube-api-access-c64rv\" (UniqueName: \"kubernetes.io/projected/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-kube-api-access-c64rv\") pod \"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6\" (UID: \"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6\") "
Oct 30 01:23:27.675775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:27.675632    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-maas-api-xks-ca\") pod \"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6\" (UID: \"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6\") "
Oct 30 01:23:27.675829 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:27.675811    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-maas-api-openshift-ca\") pod \"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6\" (UID: \"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6\") "
Oct 30 01:23:27.676236 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:27.676199    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-maas-api-openshift-ca" pod "4cf5c354-53f3-4169-8c2b-42ea97f1b2a6" (UID: "4cf5c354-53f3-4169-8c2b-42ea97f1b2a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:27.676486 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:27.676467    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:27.677770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:27.677746    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-maas-api-xks-ca" pod "4cf5c354-53f3-4169-8c2b-42ea97f1b2a6" (UID: "4cf5c354-53f3-4169-8c2b-42ea97f1b2a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:27.677876 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:27.677856    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-kube-api-access-c64rv" pod "4cf5c354-53f3-4169-8c2b-42ea97f1b2a6" (UID: "4cf5c354-53f3-4169-8c2b-42ea97f1b2a6"). InnerVolumeSpecName "kube-api-access-c64rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:27.777403 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:27.777375    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:27.777403 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:27.777399    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c64rv\" (UniqueName: \"kubernetes.io/projected/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6-kube-api-access-c64rv\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:28.429783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:28.429750    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f" event={"ID":"4cf5c354-53f3-4169-8c2b-42ea97f1b2a6","Type":"ContainerDied","Data":"6df0ca14f8f8195b0c5a661872c49608c0b6d454cc2dc827e28e0ac8e79a073e"}
Oct 30 01:23:28.429783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:28.429779    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f"
Oct 30 01:23:28.430008 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:28.429803    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6df0ca14f8f8195b0c5a661872c49608c0b6d454cc2dc827e28e0ac8e79a073e"
Oct 30 01:23:28.499961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:28.499924    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:23:31.048234 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.048194    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4" podUID="293c28e2-6799-412b-9373-1e418269cd10" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:31.453623 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.453591    6099 generic.go:356] "Generic (PLEG): container finished" podID="e5edebd8-443c-42d5-afc8-2c5495b4af01" containerID="96c5b021e92d1ef9159f2cad45b6085bbd0fe0bc3277f456b8f35733b29fb1d6" exitCode=0
Oct 30 01:23:31.453791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.453657    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp" event={"ID":"e5edebd8-443c-42d5-afc8-2c5495b4af01","Type":"ContainerDied","Data":"96c5b021e92d1ef9159f2cad45b6085bbd0fe0bc3277f456b8f35733b29fb1d6"}
Oct 30 01:23:31.453958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.453928    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp" event={"ID":"e5edebd8-443c-42d5-afc8-2c5495b4af01","Type":"ContainerDied","Data":"c6ee597299d9035bac9a52fea0b3dcf10160be2e1760b4fa389e75555094b6e7"}
Oct 30 01:23:31.454075 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.453972    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6ee597299d9035bac9a52fea0b3dcf10160be2e1760b4fa389e75555094b6e7"
Oct 30 01:23:31.472984 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.472962    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 01:23:31.526064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.526042    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-token\" (UniqueName: \"kubernetes.io/projected/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-token\") pod \"e5edebd8-443c-42d5-afc8-2c5495b4af01\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") "
Oct 30 01:23:31.526202 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.526072    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5edebd8-443c-42d5-afc8-2c5495b4af01-kube-api-access-7km9z\" (UniqueName: \"kubernetes.io/projected/e5edebd8-443c-42d5-afc8-2c5495b4af01-kube-api-access-7km9z\") pod \"e5edebd8-443c-42d5-afc8-2c5495b4af01\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") "
Oct 30 01:23:31.526202 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.526124    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-podinfo\") pod \"e5edebd8-443c-42d5-afc8-2c5495b4af01\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") "
Oct 30 01:23:31.526202 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.526190    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-workload-socket\") pod \"e5edebd8-443c-42d5-afc8-2c5495b4af01\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") "
Oct 30 01:23:31.526361 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.526265    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5edebd8-443c-42d5-afc8-2c5495b4af01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5edebd8-443c-42d5-afc8-2c5495b4af01-istiod-ca-cert\") pod \"e5edebd8-443c-42d5-afc8-2c5495b4af01\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") "
Oct 30 01:23:31.526361 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.526346    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-workload-certs\") pod \"e5edebd8-443c-42d5-afc8-2c5495b4af01\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") "
Oct 30 01:23:31.526485 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.526399    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-credential-socket\") pod \"e5edebd8-443c-42d5-afc8-2c5495b4af01\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") "
Oct 30 01:23:31.526485 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.526434    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-envoy\") pod \"e5edebd8-443c-42d5-afc8-2c5495b4af01\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") "
Oct 30 01:23:31.526596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.526516    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-data\") pod \"e5edebd8-443c-42d5-afc8-2c5495b4af01\" (UID: \"e5edebd8-443c-42d5-afc8-2c5495b4af01\") "
Oct 30 01:23:31.526596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.526581    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-workload-socket" pod "e5edebd8-443c-42d5-afc8-2c5495b4af01" (UID: "e5edebd8-443c-42d5-afc8-2c5495b4af01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:31.526746 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.526721    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-workload-certs" pod "e5edebd8-443c-42d5-afc8-2c5495b4af01" (UID: "e5edebd8-443c-42d5-afc8-2c5495b4af01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:31.526819 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.526751    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5edebd8-443c-42d5-afc8-2c5495b4af01-istiod-ca-cert" pod "e5edebd8-443c-42d5-afc8-2c5495b4af01" (UID: "e5edebd8-443c-42d5-afc8-2c5495b4af01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:31.526931 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.526911    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-credential-socket" pod "e5edebd8-443c-42d5-afc8-2c5495b4af01" (UID: "e5edebd8-443c-42d5-afc8-2c5495b4af01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:31.527301 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.527048    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-data" pod "e5edebd8-443c-42d5-afc8-2c5495b4af01" (UID: "e5edebd8-443c-42d5-afc8-2c5495b4af01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:31.527301 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.527213    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.527301 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.527231    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5edebd8-443c-42d5-afc8-2c5495b4af01-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.527301 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.527245    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.527301 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.527258    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.527301 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.527271    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.529154 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.529115    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5edebd8-443c-42d5-afc8-2c5495b4af01-kube-api-access-7km9z" pod "e5edebd8-443c-42d5-afc8-2c5495b4af01" (UID: "e5edebd8-443c-42d5-afc8-2c5495b4af01"). InnerVolumeSpecName "kube-api-access-7km9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:31.529268 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.529206    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-token" pod "e5edebd8-443c-42d5-afc8-2c5495b4af01" (UID: "e5edebd8-443c-42d5-afc8-2c5495b4af01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:31.529480 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.529463    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-envoy" pod "e5edebd8-443c-42d5-afc8-2c5495b4af01" (UID: "e5edebd8-443c-42d5-afc8-2c5495b4af01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:31.529833 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.529803    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-podinfo" pod "e5edebd8-443c-42d5-afc8-2c5495b4af01" (UID: "e5edebd8-443c-42d5-afc8-2c5495b4af01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:31.628531 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.628443    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.628531 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.628483    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.628531 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.628492    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7km9z\" (UniqueName: \"kubernetes.io/projected/e5edebd8-443c-42d5-afc8-2c5495b4af01-kube-api-access-7km9z\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.628531 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:31.628502    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5edebd8-443c-42d5-afc8-2c5495b4af01-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:32.284762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.274127    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"]
Oct 30 01:23:32.284762 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.274343    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx" podUID="1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" containerName="istio-proxy" containerID="cri-o://ca0f9be4df1c8cc2e190e17c568dad6b0747a4c6e8019294978af8d851d649c9" gracePeriod=30
Oct 30 01:23:32.463867 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.463832    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"
Oct 30 01:23:32.523264 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.523230    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"]
Oct 30 01:23:32.528579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.528545    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6add35-openshift-default-9c5d9d5ff-9l9cp"]
Oct 30 01:23:32.766820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.766788    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh"]
Oct 30 01:23:32.769481 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.769459    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae06fe-28lnh"]
Oct 30 01:23:32.843058 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.843013    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw"]
Oct 30 01:23:32.844397 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.844367    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e5edebd8-443c-42d5-afc8-2c5495b4af01" containerName="istio-proxy"
Oct 30 01:23:32.844397 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.844397    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5edebd8-443c-42d5-afc8-2c5495b4af01" containerName="istio-proxy"
Oct 30 01:23:32.844617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.844466    6099 cpu_manager.go:414] "containerMap: removing container" podUID="4cf5c354-53f3-4169-8c2b-42ea97f1b2a6" containerName="revoke-keys"
Oct 30 01:23:32.844617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.844533    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf5c354-53f3-4169-8c2b-42ea97f1b2a6" containerName="revoke-keys"
Oct 30 01:23:32.844748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.844739    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5edebd8-443c-42d5-afc8-2c5495b4af01" containerName="istio-proxy"
Oct 30 01:23:32.844812 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.844763    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf5c354-53f3-4169-8c2b-42ea97f1b2a6" containerName="revoke-keys"
Oct 30 01:23:32.854642 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.854613    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw"
Oct 30 01:23:32.866644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.866615    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw"]
Oct 30 01:23:32.928875 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.928847    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:23:32.929037 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.928931    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:23:32.950314 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.950288    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cktpp\" (UniqueName: \"kubernetes.io/projected/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-kube-api-access-cktpp\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw\" (UID: \"fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw"
Oct 30 01:23:32.950448 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.950428    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw\" (UID: \"fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw"
Oct 30 01:23:32.950583 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:32.950569    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw\" (UID: \"fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw"
Oct 30 01:23:33.051442 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:33.051410    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cktpp\" (UniqueName: \"kubernetes.io/projected/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-kube-api-access-cktpp\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw\" (UID: \"fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw"
Oct 30 01:23:33.051608 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:33.051546    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw\" (UID: \"fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw"
Oct 30 01:23:33.051744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:33.051699    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw\" (UID: \"fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw"
Oct 30 01:23:33.052238 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:33.052214    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw\" (UID: \"fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw"
Oct 30 01:23:33.054012 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:33.053989    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw\" (UID: \"fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw"
Oct 30 01:23:33.074448 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:33.074423    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cktpp\" (UniqueName: \"kubernetes.io/projected/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-kube-api-access-cktpp\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw\" (UID: \"fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw"
Oct 30 01:23:33.173562 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:33.173536    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw"
Oct 30 01:23:33.475991 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:33.475959    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw"]
Oct 30 01:23:33.477181 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:23:33.477155    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafeae69_e8b5_49e7_8f3a_10ca12e3c7e0.slice/crio-9cf6a6e39efade94b55bfcfa18a08b5638609d905501c6c23dc06acd5bb19916 WatchSource:0}: Error finding container 9cf6a6e39efade94b55bfcfa18a08b5638609d905501c6c23dc06acd5bb19916: Status 404 returned error can't find the container with id 9cf6a6e39efade94b55bfcfa18a08b5638609d905501c6c23dc06acd5bb19916
Oct 30 01:23:34.261882 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:34.261849    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9204283f-afa2-46de-a0c4-2fa88960ac72" path="/var/lib/kubelet/pods/9204283f-afa2-46de-a0c4-2fa88960ac72/volumes"
Oct 30 01:23:34.262351 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:34.262337    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5edebd8-443c-42d5-afc8-2c5495b4af01" path="/var/lib/kubelet/pods/e5edebd8-443c-42d5-afc8-2c5495b4af01/volumes"
Oct 30 01:23:34.486604 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:34.486574    6099 generic.go:356] "Generic (PLEG): container finished" podID="fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0" containerID="aa25e931f4ba953eea471689851f1973958350a700b83589953a9e00ac00b2b3" exitCode=0
Oct 30 01:23:34.487026 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:34.486637    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw" event={"ID":"fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0","Type":"ContainerDied","Data":"aa25e931f4ba953eea471689851f1973958350a700b83589953a9e00ac00b2b3"}
Oct 30 01:23:34.487026 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:34.486694    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw" event={"ID":"fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0","Type":"ContainerStarted","Data":"9cf6a6e39efade94b55bfcfa18a08b5638609d905501c6c23dc06acd5bb19916"}
Oct 30 01:23:35.638091 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:35.638068    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw"
Oct 30 01:23:35.689832 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:35.689807    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-maas-api-openshift-ca\") pod \"fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0\" (UID: \"fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0\") "
Oct 30 01:23:35.690007 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:35.689858    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-maas-api-xks-ca\") pod \"fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0\" (UID: \"fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0\") "
Oct 30 01:23:35.690007 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:35.689895    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-kube-api-access-cktpp\" (UniqueName: \"kubernetes.io/projected/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-kube-api-access-cktpp\") pod \"fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0\" (UID: \"fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0\") "
Oct 30 01:23:35.690227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:35.690205    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-maas-api-openshift-ca" pod "fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0" (UID: "fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:35.690738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:35.690717    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:35.692185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:35.692160    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-maas-api-xks-ca" pod "fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0" (UID: "fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:35.692514 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:35.692496    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-kube-api-access-cktpp" pod "fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0" (UID: "fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0"). InnerVolumeSpecName "kube-api-access-cktpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:35.792173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:35.792144    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:35.792320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:35.792175    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cktpp\" (UniqueName: \"kubernetes.io/projected/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0-kube-api-access-cktpp\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:36.501983 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:36.501951    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw"
Oct 30 01:23:36.501983 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:36.501966    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw" event={"ID":"fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0","Type":"ContainerDied","Data":"9cf6a6e39efade94b55bfcfa18a08b5638609d905501c6c23dc06acd5bb19916"}
Oct 30 01:23:36.502210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:36.502011    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cf6a6e39efade94b55bfcfa18a08b5638609d905501c6c23dc06acd5bb19916"
Oct 30 01:23:37.313862 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:23:37.313835    6099 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1acc2b4d_a946_42e3_aacd_4519ad6bd9e6.slice/crio-ca0f9be4df1c8cc2e190e17c568dad6b0747a4c6e8019294978af8d851d649c9.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 01:23:37.513853 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.513804    6099 generic.go:356] "Generic (PLEG): container finished" podID="1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" containerID="ca0f9be4df1c8cc2e190e17c568dad6b0747a4c6e8019294978af8d851d649c9" exitCode=0
Oct 30 01:23:37.514046 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.513889    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx" event={"ID":"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6","Type":"ContainerDied","Data":"ca0f9be4df1c8cc2e190e17c568dad6b0747a4c6e8019294978af8d851d649c9"}
Oct 30 01:23:37.514046 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.513966    6099 scope.go:122] "RemoveContainer" containerID="2aab72d2a83ca606bff17e89d4f1ce26a87b69cd92b97c409d5e9eed830d959c"
Oct 30 01:23:37.564305 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.564253    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 01:23:37.616425 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.616393    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-credential-socket\") pod \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") "
Oct 30 01:23:37.616564 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.616544    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-podinfo\") pod \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") "
Oct 30 01:23:37.616615 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.616606    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-data\") pod \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") "
Oct 30 01:23:37.616665 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.616634    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-envoy\") pod \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") "
Oct 30 01:23:37.616665 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.616647    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-credential-socket" pod "1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" (UID: "1acc2b4d-a946-42e3-aacd-4519ad6bd9e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:37.616815 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.616682    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-kube-api-access-t5pst\" (UniqueName: \"kubernetes.io/projected/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-kube-api-access-t5pst\") pod \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") "
Oct 30 01:23:37.616878 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.616812    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istiod-ca-cert\") pod \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") "
Oct 30 01:23:37.616940 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.616912    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-token\" (UniqueName: \"kubernetes.io/projected/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-token\") pod \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") "
Oct 30 01:23:37.617014 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.616932    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-data" pod "1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" (UID: "1acc2b4d-a946-42e3-aacd-4519ad6bd9e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:37.617014 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.616978    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-workload-certs\") pod \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") "
Oct 30 01:23:37.617136 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.617039    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-workload-socket\") pod \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\" (UID: \"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6\") "
Oct 30 01:23:37.617299 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.617260    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-workload-certs" pod "1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" (UID: "1acc2b4d-a946-42e3-aacd-4519ad6bd9e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:37.617508 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.617486    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-workload-socket" pod "1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" (UID: "1acc2b4d-a946-42e3-aacd-4519ad6bd9e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:37.617855 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.617834    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.617936 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.617860    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.617936 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.617875    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.617936 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.617889    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.618194 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.618172    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istiod-ca-cert" pod "1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" (UID: "1acc2b4d-a946-42e3-aacd-4519ad6bd9e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:37.619223 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.619088    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-kube-api-access-t5pst" pod "1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" (UID: "1acc2b4d-a946-42e3-aacd-4519ad6bd9e6"). InnerVolumeSpecName "kube-api-access-t5pst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:37.619223 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.619192    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-podinfo" pod "1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" (UID: "1acc2b4d-a946-42e3-aacd-4519ad6bd9e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:37.619394 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.619370    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-token" pod "1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" (UID: "1acc2b4d-a946-42e3-aacd-4519ad6bd9e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:37.619476 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.619459    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-envoy" pod "1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" (UID: "1acc2b4d-a946-42e3-aacd-4519ad6bd9e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:37.719064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.719033    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.719064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.719061    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5pst\" (UniqueName: \"kubernetes.io/projected/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-kube-api-access-t5pst\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.719064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.719072    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.719296 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.719081    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.719296 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:37.719089    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:38.523375 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:38.523292    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"
Oct 30 01:23:38.523375 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:38.523329    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx" event={"ID":"1acc2b4d-a946-42e3-aacd-4519ad6bd9e6","Type":"ContainerDied","Data":"fb68f9e3394dde4dc185a6013388a31e422dc76c271a6d1c117aed8ac6add4f0"}
Oct 30 01:23:38.523895 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:38.523400    6099 scope.go:122] "RemoveContainer" containerID="ca0f9be4df1c8cc2e190e17c568dad6b0747a4c6e8019294978af8d851d649c9"
Oct 30 01:23:38.577275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:38.577247    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"]
Oct 30 01:23:38.589224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:38.589198    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ea2547-openshift-default-d6bcbb6d7-vwhcx"]
Oct 30 01:23:40.080479 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.080444    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"]
Oct 30 01:23:40.081227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.080699    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp" podUID="51be8bc2-5d53-49d4-8ae0-cd281d8d511c" containerName="istio-proxy" containerID="cri-o://214e6784288ec9a99a85b4c186acad0137f01d81f5d9afadc5419f36905ea5f6" gracePeriod=30
Oct 30 01:23:40.090348 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.090323    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"]
Oct 30 01:23:40.090535 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.090516    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4" podUID="67037690-1455-44ab-aef7-8adb912e3077" containerName="istio-proxy" containerID="cri-o://805abbd20a47897e469ce77c73c514d064348ddb338cb88292f7f9a990d8741c" gracePeriod=30
Oct 30 01:23:40.262198 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.262166    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" path="/var/lib/kubelet/pods/1acc2b4d-a946-42e3-aacd-4519ad6bd9e6/volumes"
Oct 30 01:23:40.781754 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.781693    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l"]
Oct 30 01:23:40.782315 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.782298    6099 cpu_manager.go:414] "containerMap: removing container" podUID="fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0" containerName="revoke-keys"
Oct 30 01:23:40.782379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.782317    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0" containerName="revoke-keys"
Oct 30 01:23:40.782379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.782329    6099 cpu_manager.go:414] "containerMap: removing container" podUID="1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" containerName="istio-proxy"
Oct 30 01:23:40.782379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.782335    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" containerName="istio-proxy"
Oct 30 01:23:40.782379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.782370    6099 cpu_manager.go:414] "containerMap: removing container" podUID="1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" containerName="istio-proxy"
Oct 30 01:23:40.782379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.782374    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" containerName="istio-proxy"
Oct 30 01:23:40.782540 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.782471    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" containerName="istio-proxy"
Oct 30 01:23:40.782540 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.782483    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0" containerName="revoke-keys"
Oct 30 01:23:40.787286 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.787267    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l"
Oct 30 01:23:40.805479 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.805437    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:23:40.816856 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.816834    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:23:40.821659 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.821630    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l"]
Oct 30 01:23:40.863638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.863608    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcgq8\" (UniqueName: \"kubernetes.io/projected/bbbf472f-6404-4020-8925-3413535f8207-kube-api-access-vcgq8\") pod \"maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l\" (UID: \"bbbf472f-6404-4020-8925-3413535f8207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l"
Oct 30 01:23:40.863840 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.863762    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbf472f-6404-4020-8925-3413535f8207-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l\" (UID: \"bbbf472f-6404-4020-8925-3413535f8207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l"
Oct 30 01:23:40.863909 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.863849    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbf472f-6404-4020-8925-3413535f8207-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l\" (UID: \"bbbf472f-6404-4020-8925-3413535f8207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l"
Oct 30 01:23:40.965452 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.965410    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcgq8\" (UniqueName: \"kubernetes.io/projected/bbbf472f-6404-4020-8925-3413535f8207-kube-api-access-vcgq8\") pod \"maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l\" (UID: \"bbbf472f-6404-4020-8925-3413535f8207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l"
Oct 30 01:23:40.965620 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.965531    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbf472f-6404-4020-8925-3413535f8207-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l\" (UID: \"bbbf472f-6404-4020-8925-3413535f8207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l"
Oct 30 01:23:40.965620 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.965595    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbf472f-6404-4020-8925-3413535f8207-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l\" (UID: \"bbbf472f-6404-4020-8925-3413535f8207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l"
Oct 30 01:23:40.966297 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.966271    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbf472f-6404-4020-8925-3413535f8207-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l\" (UID: \"bbbf472f-6404-4020-8925-3413535f8207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l"
Oct 30 01:23:40.967883 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.967862    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbf472f-6404-4020-8925-3413535f8207-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l\" (UID: \"bbbf472f-6404-4020-8925-3413535f8207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l"
Oct 30 01:23:40.988203 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:40.988180    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcgq8\" (UniqueName: \"kubernetes.io/projected/bbbf472f-6404-4020-8925-3413535f8207-kube-api-access-vcgq8\") pod \"maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l\" (UID: \"bbbf472f-6404-4020-8925-3413535f8207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l"
Oct 30 01:23:41.048112 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:41.048080    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4" podUID="293c28e2-6799-412b-9373-1e418269cd10" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:41.101328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:41.101302    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l"
Oct 30 01:23:41.401770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:41.401742    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l"]
Oct 30 01:23:41.402659 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:23:41.402631    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbf472f_6404_4020_8925_3413535f8207.slice/crio-82313c77ad6bd5dcb206249074d78522cb71247cf921e30733781c45f56f7b04 WatchSource:0}: Error finding container 82313c77ad6bd5dcb206249074d78522cb71247cf921e30733781c45f56f7b04: Status 404 returned error can't find the container with id 82313c77ad6bd5dcb206249074d78522cb71247cf921e30733781c45f56f7b04
Oct 30 01:23:41.546032 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:41.546000    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l" event={"ID":"bbbf472f-6404-4020-8925-3413535f8207","Type":"ContainerStarted","Data":"82313c77ad6bd5dcb206249074d78522cb71247cf921e30733781c45f56f7b04"}
Oct 30 01:23:41.938224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:41.938199    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4"
Oct 30 01:23:42.106736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.106690    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293c28e2-6799-412b-9373-1e418269cd10-kube-api-access-9ng8r\" (UniqueName: \"kubernetes.io/projected/293c28e2-6799-412b-9373-1e418269cd10-kube-api-access-9ng8r\") pod \"293c28e2-6799-412b-9373-1e418269cd10\" (UID: \"293c28e2-6799-412b-9373-1e418269cd10\") "
Oct 30 01:23:42.107138 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.106900    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/293c28e2-6799-412b-9373-1e418269cd10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293c28e2-6799-412b-9373-1e418269cd10-plugins-config-volume\") pod \"293c28e2-6799-412b-9373-1e418269cd10\" (UID: \"293c28e2-6799-412b-9373-1e418269cd10\") "
Oct 30 01:23:42.107253 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.107231    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293c28e2-6799-412b-9373-1e418269cd10-plugins-config-volume" pod "293c28e2-6799-412b-9373-1e418269cd10" (UID: "293c28e2-6799-412b-9373-1e418269cd10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:42.108718 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.108683    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293c28e2-6799-412b-9373-1e418269cd10-kube-api-access-9ng8r" pod "293c28e2-6799-412b-9373-1e418269cd10" (UID: "293c28e2-6799-412b-9373-1e418269cd10"). InnerVolumeSpecName "kube-api-access-9ng8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:42.208757 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.208672    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ng8r\" (UniqueName: \"kubernetes.io/projected/293c28e2-6799-412b-9373-1e418269cd10-kube-api-access-9ng8r\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:42.208757 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.208699    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293c28e2-6799-412b-9373-1e418269cd10-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:42.464606 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.464543    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c"]
Oct 30 01:23:42.465152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.465137    6099 cpu_manager.go:414] "containerMap: removing container" podUID="293c28e2-6799-412b-9373-1e418269cd10" containerName="payload-pre-processing"
Oct 30 01:23:42.465208 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.465154    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293c28e2-6799-412b-9373-1e418269cd10" containerName="payload-pre-processing"
Oct 30 01:23:42.465289 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.465280    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1acc2b4d-a946-42e3-aacd-4519ad6bd9e6" containerName="istio-proxy"
Oct 30 01:23:42.465326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.465299    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="293c28e2-6799-412b-9373-1e418269cd10" containerName="payload-pre-processing"
Oct 30 01:23:42.468501 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.468484    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c"
Oct 30 01:23:42.490688 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.490661    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c"]
Oct 30 01:23:42.496412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.496369    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3e9921-dockercfg-48d8b\""
Oct 30 01:23:42.496559 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.496432    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3e9921\""
Oct 30 01:23:42.554901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.554870    6099 generic.go:356] "Generic (PLEG): container finished" podID="bbbf472f-6404-4020-8925-3413535f8207" containerID="f41dbcb82599c842b56a04742d1afaf1ea674a9540016f22419d78183cc7e9be" exitCode=0
Oct 30 01:23:42.555054 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.554947    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l" event={"ID":"bbbf472f-6404-4020-8925-3413535f8207","Type":"ContainerDied","Data":"f41dbcb82599c842b56a04742d1afaf1ea674a9540016f22419d78183cc7e9be"}
Oct 30 01:23:42.556294 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.556269    6099 generic.go:356] "Generic (PLEG): container finished" podID="293c28e2-6799-412b-9373-1e418269cd10" containerID="813f5334bfbabc156b72150f61202f621b6a85a7b2dedab9119f67eec7415de8" exitCode=0
Oct 30 01:23:42.556412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.556310    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4"
Oct 30 01:23:42.556412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.556348    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4" event={"ID":"293c28e2-6799-412b-9373-1e418269cd10","Type":"ContainerDied","Data":"813f5334bfbabc156b72150f61202f621b6a85a7b2dedab9119f67eec7415de8"}
Oct 30 01:23:42.556412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.556394    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4" event={"ID":"293c28e2-6799-412b-9373-1e418269cd10","Type":"ContainerDied","Data":"4ac632259085644ebe659bc5fc91183194f3066279bb368d6b506f8f18b2a35c"}
Oct 30 01:23:42.556582 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.556432    6099 scope.go:122] "RemoveContainer" containerID="813f5334bfbabc156b72150f61202f621b6a85a7b2dedab9119f67eec7415de8"
Oct 30 01:23:42.569295 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.569280    6099 scope.go:122] "RemoveContainer" containerID="813f5334bfbabc156b72150f61202f621b6a85a7b2dedab9119f67eec7415de8"
Oct 30 01:23:42.569537 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:23:42.569515    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"813f5334bfbabc156b72150f61202f621b6a85a7b2dedab9119f67eec7415de8\": container with ID starting with 813f5334bfbabc156b72150f61202f621b6a85a7b2dedab9119f67eec7415de8 not found: ID does not exist" containerID="813f5334bfbabc156b72150f61202f621b6a85a7b2dedab9119f67eec7415de8"
Oct 30 01:23:42.569614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.569547    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813f5334bfbabc156b72150f61202f621b6a85a7b2dedab9119f67eec7415de8"} err="failed to get container status \"813f5334bfbabc156b72150f61202f621b6a85a7b2dedab9119f67eec7415de8\": rpc error: code = NotFound desc = could not find container \"813f5334bfbabc156b72150f61202f621b6a85a7b2dedab9119f67eec7415de8\": container with ID starting with 813f5334bfbabc156b72150f61202f621b6a85a7b2dedab9119f67eec7415de8 not found: ID does not exist"
Oct 30 01:23:42.614410 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.614382    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48894977-815d-4b0d-be69-c9d7f29aa1a4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c\" (UID: \"48894977-815d-4b0d-be69-c9d7f29aa1a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c"
Oct 30 01:23:42.614552 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.614468    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwp2m\" (UniqueName: \"kubernetes.io/projected/48894977-815d-4b0d-be69-c9d7f29aa1a4-kube-api-access-rwp2m\") pod \"payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c\" (UID: \"48894977-815d-4b0d-be69-c9d7f29aa1a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c"
Oct 30 01:23:42.631594 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.631571    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4"]
Oct 30 01:23:42.634673 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.634651    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae06fe-96bfc5f55-4fjl4"]
Oct 30 01:23:42.716141 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.716076    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48894977-815d-4b0d-be69-c9d7f29aa1a4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c\" (UID: \"48894977-815d-4b0d-be69-c9d7f29aa1a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c"
Oct 30 01:23:42.716246 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.716158    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwp2m\" (UniqueName: \"kubernetes.io/projected/48894977-815d-4b0d-be69-c9d7f29aa1a4-kube-api-access-rwp2m\") pod \"payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c\" (UID: \"48894977-815d-4b0d-be69-c9d7f29aa1a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c"
Oct 30 01:23:42.716835 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.716815    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48894977-815d-4b0d-be69-c9d7f29aa1a4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c\" (UID: \"48894977-815d-4b0d-be69-c9d7f29aa1a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c"
Oct 30 01:23:42.739428 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.739395    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwp2m\" (UniqueName: \"kubernetes.io/projected/48894977-815d-4b0d-be69-c9d7f29aa1a4-kube-api-access-rwp2m\") pod \"payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c\" (UID: \"48894977-815d-4b0d-be69-c9d7f29aa1a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c"
Oct 30 01:23:42.821101 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:42.821073    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c"
Oct 30 01:23:43.127249 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:43.127219    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c"]
Oct 30 01:23:43.138877 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:23:43.137878    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48894977_815d_4b0d_be69_c9d7f29aa1a4.slice/crio-e3d2f8e6b5a065a4770f8239ad68920470c094c67f22ff83ab40948c3a7f17ba WatchSource:0}: Error finding container e3d2f8e6b5a065a4770f8239ad68920470c094c67f22ff83ab40948c3a7f17ba: Status 404 returned error can't find the container with id e3d2f8e6b5a065a4770f8239ad68920470c094c67f22ff83ab40948c3a7f17ba
Oct 30 01:23:43.303197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:43.303165    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95"]
Oct 30 01:23:43.305592 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:43.305556    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal" podUID="f8c3a9edbd8cbf03175beab8c4db7242" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 01:23:43.566435 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:43.566404    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c" event={"ID":"48894977-815d-4b0d-be69-c9d7f29aa1a4","Type":"ContainerStarted","Data":"c3db3e0512be4f1670bbe0e98ad3efa0bb701550fc9f792a6ecb2b02d56dd3db"}
Oct 30 01:23:43.566625 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:43.566462    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c" event={"ID":"48894977-815d-4b0d-be69-c9d7f29aa1a4","Type":"ContainerStarted","Data":"e3d2f8e6b5a065a4770f8239ad68920470c094c67f22ff83ab40948c3a7f17ba"}
Oct 30 01:23:43.566753 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:43.566718    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c"
Oct 30 01:23:43.608404 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:43.608348    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c" podStartSLOduration=1.6083340320000001 podStartE2EDuration="1.608334032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:43.606230343 +0000 UTC m=+2917.463138100" watchObservedRunningTime="2026-10-30 01:23:43.608334032 +0000 UTC m=+2917.465241787"
Oct 30 01:23:43.749838 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:43.749814    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l"
Oct 30 01:23:43.933785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:43.933237    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbf472f-6404-4020-8925-3413535f8207-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbf472f-6404-4020-8925-3413535f8207-maas-api-openshift-ca\") pod \"bbbf472f-6404-4020-8925-3413535f8207\" (UID: \"bbbf472f-6404-4020-8925-3413535f8207\") "
Oct 30 01:23:43.933785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:43.933319    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbf472f-6404-4020-8925-3413535f8207-kube-api-access-vcgq8\" (UniqueName: \"kubernetes.io/projected/bbbf472f-6404-4020-8925-3413535f8207-kube-api-access-vcgq8\") pod \"bbbf472f-6404-4020-8925-3413535f8207\" (UID: \"bbbf472f-6404-4020-8925-3413535f8207\") "
Oct 30 01:23:43.933785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:43.933572    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbbf472f-6404-4020-8925-3413535f8207-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbf472f-6404-4020-8925-3413535f8207-maas-api-xks-ca\") pod \"bbbf472f-6404-4020-8925-3413535f8207\" (UID: \"bbbf472f-6404-4020-8925-3413535f8207\") "
Oct 30 01:23:43.935746 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:43.935692    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbf472f-6404-4020-8925-3413535f8207-maas-api-openshift-ca" pod "bbbf472f-6404-4020-8925-3413535f8207" (UID: "bbbf472f-6404-4020-8925-3413535f8207"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:43.949726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:43.949357    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbf472f-6404-4020-8925-3413535f8207-kube-api-access-vcgq8" pod "bbbf472f-6404-4020-8925-3413535f8207" (UID: "bbbf472f-6404-4020-8925-3413535f8207"). InnerVolumeSpecName "kube-api-access-vcgq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:43.949726 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:43.949463    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbbf472f-6404-4020-8925-3413535f8207-maas-api-xks-ca" pod "bbbf472f-6404-4020-8925-3413535f8207" (UID: "bbbf472f-6404-4020-8925-3413535f8207"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:44.035471 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:44.035437    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbf472f-6404-4020-8925-3413535f8207-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:44.035471 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:44.035468    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcgq8\" (UniqueName: \"kubernetes.io/projected/bbbf472f-6404-4020-8925-3413535f8207-kube-api-access-vcgq8\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:44.035471 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:44.035477    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbf472f-6404-4020-8925-3413535f8207-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:44.261768 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:44.261687    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="293c28e2-6799-412b-9373-1e418269cd10" path="/var/lib/kubelet/pods/293c28e2-6799-412b-9373-1e418269cd10/volumes"
Oct 30 01:23:44.515554 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:44.515472    6099 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:23:44.515554 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:44.515529    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4" podUID="67037690-1455-44ab-aef7-8adb912e3077" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:23:44.574284 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:44.574255    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l"
Oct 30 01:23:44.574449 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:44.574289    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l" event={"ID":"bbbf472f-6404-4020-8925-3413535f8207","Type":"ContainerDied","Data":"82313c77ad6bd5dcb206249074d78522cb71247cf921e30733781c45f56f7b04"}
Oct 30 01:23:44.574449 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:44.574337    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82313c77ad6bd5dcb206249074d78522cb71247cf921e30733781c45f56f7b04"
Oct 30 01:23:45.383318 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.383224    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:23:45.406600 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.406579    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:23:45.453152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.453118    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-podinfo\") pod \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") "
Oct 30 01:23:45.453331 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.453174    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-token\" (UniqueName: \"kubernetes.io/projected/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-token\") pod \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") "
Oct 30 01:23:45.453331 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.453237    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-workload-socket\") pod \"67037690-1455-44ab-aef7-8adb912e3077\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") "
Oct 30 01:23:45.453331 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.453271    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-workload-certs\") pod \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") "
Oct 30 01:23:45.453331 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.453321    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67037690-1455-44ab-aef7-8adb912e3077-istio-token\" (UniqueName: \"kubernetes.io/projected/67037690-1455-44ab-aef7-8adb912e3077-istio-token\") pod \"67037690-1455-44ab-aef7-8adb912e3077\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") "
Oct 30 01:23:45.453585 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.453397    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-credential-socket\") pod \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") "
Oct 30 01:23:45.453585 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.453425    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-workload-socket\") pod \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") "
Oct 30 01:23:45.453585 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.453468    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-data\") pod \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") "
Oct 30 01:23:45.453585 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.453521    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-workload-socket" pod "67037690-1455-44ab-aef7-8adb912e3077" (UID: "67037690-1455-44ab-aef7-8adb912e3077"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:45.453585 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.453553    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-kube-api-access-slmf4\" (UniqueName: \"kubernetes.io/projected/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-kube-api-access-slmf4\") pod \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") "
Oct 30 01:23:45.454000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.453613    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-credential-socket\") pod \"67037690-1455-44ab-aef7-8adb912e3077\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") "
Oct 30 01:23:45.454000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.453657    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istiod-ca-cert\") pod \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") "
Oct 30 01:23:45.454000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.453744    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-credential-socket" pod "51be8bc2-5d53-49d4-8ae0-cd281d8d511c" (UID: "51be8bc2-5d53-49d4-8ae0-cd281d8d511c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:45.454000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.453752    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-workload-certs\") pod \"67037690-1455-44ab-aef7-8adb912e3077\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") "
Oct 30 01:23:45.454000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.453820    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67037690-1455-44ab-aef7-8adb912e3077-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67037690-1455-44ab-aef7-8adb912e3077-istiod-ca-cert\") pod \"67037690-1455-44ab-aef7-8adb912e3077\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") "
Oct 30 01:23:45.454000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.453878    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67037690-1455-44ab-aef7-8adb912e3077-kube-api-access-mxcxf\" (UniqueName: \"kubernetes.io/projected/67037690-1455-44ab-aef7-8adb912e3077-kube-api-access-mxcxf\") pod \"67037690-1455-44ab-aef7-8adb912e3077\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") "
Oct 30 01:23:45.454000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.453957    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-istio-data\") pod \"67037690-1455-44ab-aef7-8adb912e3077\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") "
Oct 30 01:23:45.454000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.454002    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-istio-envoy\") pod \"67037690-1455-44ab-aef7-8adb912e3077\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") "
Oct 30 01:23:45.454436 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.454017    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-workload-certs" pod "67037690-1455-44ab-aef7-8adb912e3077" (UID: "67037690-1455-44ab-aef7-8adb912e3077"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:45.454436 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.454045    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-envoy\") pod \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\" (UID: \"51be8bc2-5d53-49d4-8ae0-cd281d8d511c\") "
Oct 30 01:23:45.454436 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.454074    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67037690-1455-44ab-aef7-8adb912e3077-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67037690-1455-44ab-aef7-8adb912e3077-istio-podinfo\") pod \"67037690-1455-44ab-aef7-8adb912e3077\" (UID: \"67037690-1455-44ab-aef7-8adb912e3077\") "
Oct 30 01:23:45.454436 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.454210    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-workload-socket" pod "51be8bc2-5d53-49d4-8ae0-cd281d8d511c" (UID: "51be8bc2-5d53-49d4-8ae0-cd281d8d511c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:45.454436 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.454383    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-data" pod "51be8bc2-5d53-49d4-8ae0-cd281d8d511c" (UID: "51be8bc2-5d53-49d4-8ae0-cd281d8d511c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:45.454768 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.454745    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.454918 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.454900    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.455019 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.455006    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.455122 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.455110    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.455223 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.455211    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.455678 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.455654    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istiod-ca-cert" pod "51be8bc2-5d53-49d4-8ae0-cd281d8d511c" (UID: "51be8bc2-5d53-49d4-8ae0-cd281d8d511c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:45.455926 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.455898    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-credential-socket" pod "67037690-1455-44ab-aef7-8adb912e3077" (UID: "67037690-1455-44ab-aef7-8adb912e3077"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:45.456311 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.456277    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67037690-1455-44ab-aef7-8adb912e3077-istiod-ca-cert" pod "67037690-1455-44ab-aef7-8adb912e3077" (UID: "67037690-1455-44ab-aef7-8adb912e3077"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:45.457081 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.456815    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-workload-certs" pod "51be8bc2-5d53-49d4-8ae0-cd281d8d511c" (UID: "51be8bc2-5d53-49d4-8ae0-cd281d8d511c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:45.458776 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.458683    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-istio-data" pod "67037690-1455-44ab-aef7-8adb912e3077" (UID: "67037690-1455-44ab-aef7-8adb912e3077"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:45.458999 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.458975    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-podinfo" pod "51be8bc2-5d53-49d4-8ae0-cd281d8d511c" (UID: "51be8bc2-5d53-49d4-8ae0-cd281d8d511c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:45.459477 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.459453    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67037690-1455-44ab-aef7-8adb912e3077-istio-podinfo" pod "67037690-1455-44ab-aef7-8adb912e3077" (UID: "67037690-1455-44ab-aef7-8adb912e3077"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:45.459477 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.459460    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67037690-1455-44ab-aef7-8adb912e3077-kube-api-access-mxcxf" pod "67037690-1455-44ab-aef7-8adb912e3077" (UID: "67037690-1455-44ab-aef7-8adb912e3077"). InnerVolumeSpecName "kube-api-access-mxcxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:45.459646 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.459480    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67037690-1455-44ab-aef7-8adb912e3077-istio-token" pod "67037690-1455-44ab-aef7-8adb912e3077" (UID: "67037690-1455-44ab-aef7-8adb912e3077"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:45.459841 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.459814    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-istio-envoy" pod "67037690-1455-44ab-aef7-8adb912e3077" (UID: "67037690-1455-44ab-aef7-8adb912e3077"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:45.459960 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.459938    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-kube-api-access-slmf4" pod "51be8bc2-5d53-49d4-8ae0-cd281d8d511c" (UID: "51be8bc2-5d53-49d4-8ae0-cd281d8d511c"). InnerVolumeSpecName "kube-api-access-slmf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:45.460127 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.460104    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-token" pod "51be8bc2-5d53-49d4-8ae0-cd281d8d511c" (UID: "51be8bc2-5d53-49d4-8ae0-cd281d8d511c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:45.460195 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.460170    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-envoy" pod "51be8bc2-5d53-49d4-8ae0-cd281d8d511c" (UID: "51be8bc2-5d53-49d4-8ae0-cd281d8d511c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:45.556566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.556534    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.556566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.556562    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67037690-1455-44ab-aef7-8adb912e3077-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.556566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.556571    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.556822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.556581    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.556822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.556590    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.556822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.556598    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67037690-1455-44ab-aef7-8adb912e3077-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.556822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.556607    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slmf4\" (UniqueName: \"kubernetes.io/projected/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-kube-api-access-slmf4\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.556822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.556616    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.556822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.556625    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51be8bc2-5d53-49d4-8ae0-cd281d8d511c-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.556822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.556633    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67037690-1455-44ab-aef7-8adb912e3077-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.556822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.556641    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxcxf\" (UniqueName: \"kubernetes.io/projected/67037690-1455-44ab-aef7-8adb912e3077-kube-api-access-mxcxf\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.556822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.556648    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.556822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.556657    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67037690-1455-44ab-aef7-8adb912e3077-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.580261 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.580222    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"]
Oct 30 01:23:45.584597 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.584562    6099 generic.go:356] "Generic (PLEG): container finished" podID="67037690-1455-44ab-aef7-8adb912e3077" containerID="805abbd20a47897e469ce77c73c514d064348ddb338cb88292f7f9a990d8741c" exitCode=0
Oct 30 01:23:45.584768 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.584641    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"
Oct 30 01:23:45.584768 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.584646    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4" event={"ID":"67037690-1455-44ab-aef7-8adb912e3077","Type":"ContainerDied","Data":"805abbd20a47897e469ce77c73c514d064348ddb338cb88292f7f9a990d8741c"}
Oct 30 01:23:45.584768 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.584761    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4" event={"ID":"67037690-1455-44ab-aef7-8adb912e3077","Type":"ContainerDied","Data":"84368adec3e825824aaec4f9900e56db14c58974f494c8f82e97348d72133c42"}
Oct 30 01:23:45.584922 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.584793    6099 scope.go:122] "RemoveContainer" containerID="805abbd20a47897e469ce77c73c514d064348ddb338cb88292f7f9a990d8741c"
Oct 30 01:23:45.586663 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.586637    6099 generic.go:356] "Generic (PLEG): container finished" podID="51be8bc2-5d53-49d4-8ae0-cd281d8d511c" containerID="214e6784288ec9a99a85b4c186acad0137f01d81f5d9afadc5419f36905ea5f6" exitCode=0
Oct 30 01:23:45.586815 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.586674    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp" event={"ID":"51be8bc2-5d53-49d4-8ae0-cd281d8d511c","Type":"ContainerDied","Data":"214e6784288ec9a99a85b4c186acad0137f01d81f5d9afadc5419f36905ea5f6"}
Oct 30 01:23:45.586815 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.586733    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"
Oct 30 01:23:45.586815 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.586783    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp" event={"ID":"51be8bc2-5d53-49d4-8ae0-cd281d8d511c","Type":"ContainerDied","Data":"1188d18bab63dc42ab694b6faf86425b2829880c3d2d0931411f25a10832c1e3"}
Oct 30 01:23:45.591671 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.591111    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"]
Oct 30 01:23:45.599601 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.599474    6099 scope.go:122] "RemoveContainer" containerID="c5d4c46681067afaec4d5a7b7b938d43b50bba061dd223d8f1259b7e13c9e23f"
Oct 30 01:23:45.612076 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.612049    6099 scope.go:122] "RemoveContainer" containerID="805abbd20a47897e469ce77c73c514d064348ddb338cb88292f7f9a990d8741c"
Oct 30 01:23:45.612373 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:23:45.612349    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805abbd20a47897e469ce77c73c514d064348ddb338cb88292f7f9a990d8741c\": container with ID starting with 805abbd20a47897e469ce77c73c514d064348ddb338cb88292f7f9a990d8741c not found: ID does not exist" containerID="805abbd20a47897e469ce77c73c514d064348ddb338cb88292f7f9a990d8741c"
Oct 30 01:23:45.612466 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.612392    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805abbd20a47897e469ce77c73c514d064348ddb338cb88292f7f9a990d8741c"} err="failed to get container status \"805abbd20a47897e469ce77c73c514d064348ddb338cb88292f7f9a990d8741c\": rpc error: code = NotFound desc = could not find container \"805abbd20a47897e469ce77c73c514d064348ddb338cb88292f7f9a990d8741c\": container with ID starting with 805abbd20a47897e469ce77c73c514d064348ddb338cb88292f7f9a990d8741c not found: ID does not exist"
Oct 30 01:23:45.612466 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.612419    6099 scope.go:122] "RemoveContainer" containerID="c5d4c46681067afaec4d5a7b7b938d43b50bba061dd223d8f1259b7e13c9e23f"
Oct 30 01:23:45.612756 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:23:45.612698    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5d4c46681067afaec4d5a7b7b938d43b50bba061dd223d8f1259b7e13c9e23f\": container with ID starting with c5d4c46681067afaec4d5a7b7b938d43b50bba061dd223d8f1259b7e13c9e23f not found: ID does not exist" containerID="c5d4c46681067afaec4d5a7b7b938d43b50bba061dd223d8f1259b7e13c9e23f"
Oct 30 01:23:45.612874 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.612763    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5d4c46681067afaec4d5a7b7b938d43b50bba061dd223d8f1259b7e13c9e23f"} err="failed to get container status \"c5d4c46681067afaec4d5a7b7b938d43b50bba061dd223d8f1259b7e13c9e23f\": rpc error: code = NotFound desc = could not find container \"c5d4c46681067afaec4d5a7b7b938d43b50bba061dd223d8f1259b7e13c9e23f\": container with ID starting with c5d4c46681067afaec4d5a7b7b938d43b50bba061dd223d8f1259b7e13c9e23f not found: ID does not exist"
Oct 30 01:23:45.612874 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.612784    6099 scope.go:122] "RemoveContainer" containerID="214e6784288ec9a99a85b4c186acad0137f01d81f5d9afadc5419f36905ea5f6"
Oct 30 01:23:45.623362 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.623340    6099 scope.go:122] "RemoveContainer" containerID="214e6784288ec9a99a85b4c186acad0137f01d81f5d9afadc5419f36905ea5f6"
Oct 30 01:23:45.623906 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:23:45.623882    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"214e6784288ec9a99a85b4c186acad0137f01d81f5d9afadc5419f36905ea5f6\": container with ID starting with 214e6784288ec9a99a85b4c186acad0137f01d81f5d9afadc5419f36905ea5f6 not found: ID does not exist" containerID="214e6784288ec9a99a85b4c186acad0137f01d81f5d9afadc5419f36905ea5f6"
Oct 30 01:23:45.624017 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.623911    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"214e6784288ec9a99a85b4c186acad0137f01d81f5d9afadc5419f36905ea5f6"} err="failed to get container status \"214e6784288ec9a99a85b4c186acad0137f01d81f5d9afadc5419f36905ea5f6\": rpc error: code = NotFound desc = could not find container \"214e6784288ec9a99a85b4c186acad0137f01d81f5d9afadc5419f36905ea5f6\": container with ID starting with 214e6784288ec9a99a85b4c186acad0137f01d81f5d9afadc5419f36905ea5f6 not found: ID does not exist"
Oct 30 01:23:45.661294 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.661270    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"]
Oct 30 01:23:45.663994 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.663976    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-wj4kp"]
Oct 30 01:23:45.696167 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.696141    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"]
Oct 30 01:23:45.709300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:45.709264    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae06fe-openshift-default-56454b6787-8jpk4"]
Oct 30 01:23:46.262426 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:46.262390    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51be8bc2-5d53-49d4-8ae0-cd281d8d511c" path="/var/lib/kubelet/pods/51be8bc2-5d53-49d4-8ae0-cd281d8d511c/volumes"
Oct 30 01:23:46.262955 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:46.262938    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67037690-1455-44ab-aef7-8adb912e3077" path="/var/lib/kubelet/pods/67037690-1455-44ab-aef7-8adb912e3077/volumes"
Oct 30 01:23:46.631793 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:23:46.631752    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:23:48.303662 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:48.303629    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95" podUID="41feca8d-0a29-42e4-9969-c1f6e4ee7f36" containerName="payload-pre-processing" containerID="cri-o://9f63313577e881b892d25c7b079c4f25dec08298c35f2a6750860351198f2eda" gracePeriod=25
Oct 30 01:23:49.532866 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:49.532832    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b"]
Oct 30 01:23:49.533304 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:49.533022    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b" podUID="261fed99-0c9e-42cb-ad91-540d2b125576" containerName="payload-processing" containerID="cri-o://adb8cba0c97eb094676379597f401d4072f38fcf60e2da07a7b5742880cf5a9b" gracePeriod=30
Oct 30 01:23:49.812698 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:49.812658    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b"
Oct 30 01:23:49.916015 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:49.915984    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/261fed99-0c9e-42cb-ad91-540d2b125576-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261fed99-0c9e-42cb-ad91-540d2b125576-plugins-config-volume\") pod \"261fed99-0c9e-42cb-ad91-540d2b125576\" (UID: \"261fed99-0c9e-42cb-ad91-540d2b125576\") "
Oct 30 01:23:49.916197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:49.916165    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261fed99-0c9e-42cb-ad91-540d2b125576-kube-api-access-gl9fk\" (UniqueName: \"kubernetes.io/projected/261fed99-0c9e-42cb-ad91-540d2b125576-kube-api-access-gl9fk\") pod \"261fed99-0c9e-42cb-ad91-540d2b125576\" (UID: \"261fed99-0c9e-42cb-ad91-540d2b125576\") "
Oct 30 01:23:49.916438 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:49.916411    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261fed99-0c9e-42cb-ad91-540d2b125576-plugins-config-volume" pod "261fed99-0c9e-42cb-ad91-540d2b125576" (UID: "261fed99-0c9e-42cb-ad91-540d2b125576"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:49.916561 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:49.916534    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261fed99-0c9e-42cb-ad91-540d2b125576-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:49.918466 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:49.918438    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261fed99-0c9e-42cb-ad91-540d2b125576-kube-api-access-gl9fk" pod "261fed99-0c9e-42cb-ad91-540d2b125576" (UID: "261fed99-0c9e-42cb-ad91-540d2b125576"). InnerVolumeSpecName "kube-api-access-gl9fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:50.018460 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:50.018422    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl9fk\" (UniqueName: \"kubernetes.io/projected/261fed99-0c9e-42cb-ad91-540d2b125576-kube-api-access-gl9fk\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:50.629185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:50.629154    6099 generic.go:356] "Generic (PLEG): container finished" podID="261fed99-0c9e-42cb-ad91-540d2b125576" containerID="adb8cba0c97eb094676379597f401d4072f38fcf60e2da07a7b5742880cf5a9b" exitCode=0
Oct 30 01:23:50.629657 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:50.629196    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b"
Oct 30 01:23:50.629657 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:50.629229    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b" event={"ID":"261fed99-0c9e-42cb-ad91-540d2b125576","Type":"ContainerDied","Data":"adb8cba0c97eb094676379597f401d4072f38fcf60e2da07a7b5742880cf5a9b"}
Oct 30 01:23:50.629657 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:50.629301    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b" event={"ID":"261fed99-0c9e-42cb-ad91-540d2b125576","Type":"ContainerDied","Data":"cc4a73a624af7b6cde900f9ad529117a44c6023b530c037bde19052c12b9ef05"}
Oct 30 01:23:50.629657 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:50.629337    6099 scope.go:122] "RemoveContainer" containerID="adb8cba0c97eb094676379597f401d4072f38fcf60e2da07a7b5742880cf5a9b"
Oct 30 01:23:50.641981 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:50.641962    6099 scope.go:122] "RemoveContainer" containerID="adb8cba0c97eb094676379597f401d4072f38fcf60e2da07a7b5742880cf5a9b"
Oct 30 01:23:50.642260 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:23:50.642243    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb8cba0c97eb094676379597f401d4072f38fcf60e2da07a7b5742880cf5a9b\": container with ID starting with adb8cba0c97eb094676379597f401d4072f38fcf60e2da07a7b5742880cf5a9b not found: ID does not exist" containerID="adb8cba0c97eb094676379597f401d4072f38fcf60e2da07a7b5742880cf5a9b"
Oct 30 01:23:50.642311 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:50.642272    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb8cba0c97eb094676379597f401d4072f38fcf60e2da07a7b5742880cf5a9b"} err="failed to get container status \"adb8cba0c97eb094676379597f401d4072f38fcf60e2da07a7b5742880cf5a9b\": rpc error: code = NotFound desc = could not find container \"adb8cba0c97eb094676379597f401d4072f38fcf60e2da07a7b5742880cf5a9b\": container with ID starting with adb8cba0c97eb094676379597f401d4072f38fcf60e2da07a7b5742880cf5a9b not found: ID does not exist"
Oct 30 01:23:50.681083 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:50.681063    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b"]
Oct 30 01:23:50.692881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:50.692857    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abf8ae-7cccbf8ff6-b6k7b"]
Oct 30 01:23:51.771839 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.771808    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:23:51.786937 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.786906    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n"]
Oct 30 01:23:51.787516 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.787500    6099 cpu_manager.go:414] "containerMap: removing container" podUID="51be8bc2-5d53-49d4-8ae0-cd281d8d511c" containerName="istio-proxy"
Oct 30 01:23:51.787516 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.787516    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51be8bc2-5d53-49d4-8ae0-cd281d8d511c" containerName="istio-proxy"
Oct 30 01:23:51.787670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.787565    6099 cpu_manager.go:414] "containerMap: removing container" podUID="67037690-1455-44ab-aef7-8adb912e3077" containerName="istio-proxy"
Oct 30 01:23:51.787670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.787571    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67037690-1455-44ab-aef7-8adb912e3077" containerName="istio-proxy"
Oct 30 01:23:51.787670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.787581    6099 cpu_manager.go:414] "containerMap: removing container" podUID="261fed99-0c9e-42cb-ad91-540d2b125576" containerName="payload-processing"
Oct 30 01:23:51.787670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.787587    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261fed99-0c9e-42cb-ad91-540d2b125576" containerName="payload-processing"
Oct 30 01:23:51.787670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.787597    6099 cpu_manager.go:414] "containerMap: removing container" podUID="67037690-1455-44ab-aef7-8adb912e3077" containerName="istio-proxy"
Oct 30 01:23:51.787670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.787601    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67037690-1455-44ab-aef7-8adb912e3077" containerName="istio-proxy"
Oct 30 01:23:51.787670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.787607    6099 cpu_manager.go:414] "containerMap: removing container" podUID="bbbf472f-6404-4020-8925-3413535f8207" containerName="revoke-keys"
Oct 30 01:23:51.787670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.787614    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbf472f-6404-4020-8925-3413535f8207" containerName="revoke-keys"
Oct 30 01:23:51.788166 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.787720    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="67037690-1455-44ab-aef7-8adb912e3077" containerName="istio-proxy"
Oct 30 01:23:51.788166 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.787734    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="261fed99-0c9e-42cb-ad91-540d2b125576" containerName="payload-processing"
Oct 30 01:23:51.788166 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.787742    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="67037690-1455-44ab-aef7-8adb912e3077" containerName="istio-proxy"
Oct 30 01:23:51.788166 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.787748    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbf472f-6404-4020-8925-3413535f8207" containerName="revoke-keys"
Oct 30 01:23:51.788166 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.787756    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="51be8bc2-5d53-49d4-8ae0-cd281d8d511c" containerName="istio-proxy"
Oct 30 01:23:51.792563 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.792542    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n"
Oct 30 01:23:51.810863 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.810835    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:23:51.821394 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.816306    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n"]
Oct 30 01:23:51.821580 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.816504    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:23:51.845778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.845745    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kr7q\" (UniqueName: \"kubernetes.io/projected/4cacfc24-be5d-4505-a761-18aa477444fb-kube-api-access-6kr7q\") pod \"maas-api-revoke-sub-bc666335-5fq9n\" (UID: \"4cacfc24-be5d-4505-a761-18aa477444fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n"
Oct 30 01:23:51.846042 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.846019    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cacfc24-be5d-4505-a761-18aa477444fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc666335-5fq9n\" (UID: \"4cacfc24-be5d-4505-a761-18aa477444fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n"
Oct 30 01:23:51.846153 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.846134    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cacfc24-be5d-4505-a761-18aa477444fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc666335-5fq9n\" (UID: \"4cacfc24-be5d-4505-a761-18aa477444fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n"
Oct 30 01:23:51.947112 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.947080    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kr7q\" (UniqueName: \"kubernetes.io/projected/4cacfc24-be5d-4505-a761-18aa477444fb-kube-api-access-6kr7q\") pod \"maas-api-revoke-sub-bc666335-5fq9n\" (UID: \"4cacfc24-be5d-4505-a761-18aa477444fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n"
Oct 30 01:23:51.947389 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.947122    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cacfc24-be5d-4505-a761-18aa477444fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc666335-5fq9n\" (UID: \"4cacfc24-be5d-4505-a761-18aa477444fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n"
Oct 30 01:23:51.947389 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.947187    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cacfc24-be5d-4505-a761-18aa477444fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc666335-5fq9n\" (UID: \"4cacfc24-be5d-4505-a761-18aa477444fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n"
Oct 30 01:23:51.948010 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.947987    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cacfc24-be5d-4505-a761-18aa477444fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc666335-5fq9n\" (UID: \"4cacfc24-be5d-4505-a761-18aa477444fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n"
Oct 30 01:23:51.949658 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.949636    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cacfc24-be5d-4505-a761-18aa477444fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc666335-5fq9n\" (UID: \"4cacfc24-be5d-4505-a761-18aa477444fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n"
Oct 30 01:23:51.969975 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:51.969953    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kr7q\" (UniqueName: \"kubernetes.io/projected/4cacfc24-be5d-4505-a761-18aa477444fb-kube-api-access-6kr7q\") pod \"maas-api-revoke-sub-bc666335-5fq9n\" (UID: \"4cacfc24-be5d-4505-a761-18aa477444fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n"
Oct 30 01:23:52.124919 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:52.124888    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n"
Oct 30 01:23:52.261289 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:52.261258    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261fed99-0c9e-42cb-ad91-540d2b125576" path="/var/lib/kubelet/pods/261fed99-0c9e-42cb-ad91-540d2b125576/volumes"
Oct 30 01:23:52.426406 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:52.426372    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n"]
Oct 30 01:23:52.427293 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:23:52.427259    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cacfc24_be5d_4505_a761_18aa477444fb.slice/crio-e85734b8013230c1fe7a140fd5f11d5267319837be488bfca5edddd762fbf121 WatchSource:0}: Error finding container e85734b8013230c1fe7a140fd5f11d5267319837be488bfca5edddd762fbf121: Status 404 returned error can't find the container with id e85734b8013230c1fe7a140fd5f11d5267319837be488bfca5edddd762fbf121
Oct 30 01:23:52.644366 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:52.644334    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n" event={"ID":"4cacfc24-be5d-4505-a761-18aa477444fb","Type":"ContainerStarted","Data":"e85734b8013230c1fe7a140fd5f11d5267319837be488bfca5edddd762fbf121"}
Oct 30 01:23:53.655650 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:53.655565    6099 generic.go:356] "Generic (PLEG): container finished" podID="4cacfc24-be5d-4505-a761-18aa477444fb" containerID="de4480663c8df7dd8d95d6999ed086c98d1e7bde23b340610c46082236beea1a" exitCode=0
Oct 30 01:23:53.656069 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:53.655654    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n" event={"ID":"4cacfc24-be5d-4505-a761-18aa477444fb","Type":"ContainerDied","Data":"de4480663c8df7dd8d95d6999ed086c98d1e7bde23b340610c46082236beea1a"}
Oct 30 01:23:54.577367 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:54.577333    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c"
Oct 30 01:23:54.736223 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:54.736187    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95" podUID="41feca8d-0a29-42e4-9969-c1f6e4ee7f36" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:54.832629 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:54.832569    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n"
Oct 30 01:23:54.997048 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:54.997016    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cacfc24-be5d-4505-a761-18aa477444fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cacfc24-be5d-4505-a761-18aa477444fb-maas-api-xks-ca\") pod \"4cacfc24-be5d-4505-a761-18aa477444fb\" (UID: \"4cacfc24-be5d-4505-a761-18aa477444fb\") "
Oct 30 01:23:54.997234 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:54.997158    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cacfc24-be5d-4505-a761-18aa477444fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cacfc24-be5d-4505-a761-18aa477444fb-maas-api-openshift-ca\") pod \"4cacfc24-be5d-4505-a761-18aa477444fb\" (UID: \"4cacfc24-be5d-4505-a761-18aa477444fb\") "
Oct 30 01:23:54.997234 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:54.997195    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cacfc24-be5d-4505-a761-18aa477444fb-kube-api-access-6kr7q\" (UniqueName: \"kubernetes.io/projected/4cacfc24-be5d-4505-a761-18aa477444fb-kube-api-access-6kr7q\") pod \"4cacfc24-be5d-4505-a761-18aa477444fb\" (UID: \"4cacfc24-be5d-4505-a761-18aa477444fb\") "
Oct 30 01:23:54.997543 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:54.997518    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cacfc24-be5d-4505-a761-18aa477444fb-maas-api-openshift-ca" pod "4cacfc24-be5d-4505-a761-18aa477444fb" (UID: "4cacfc24-be5d-4505-a761-18aa477444fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:54.999173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:54.999143    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cacfc24-be5d-4505-a761-18aa477444fb-maas-api-xks-ca" pod "4cacfc24-be5d-4505-a761-18aa477444fb" (UID: "4cacfc24-be5d-4505-a761-18aa477444fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:54.999262 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:54.999194    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cacfc24-be5d-4505-a761-18aa477444fb-kube-api-access-6kr7q" pod "4cacfc24-be5d-4505-a761-18aa477444fb" (UID: "4cacfc24-be5d-4505-a761-18aa477444fb"). InnerVolumeSpecName "kube-api-access-6kr7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:55.098332 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:55.098276    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cacfc24-be5d-4505-a761-18aa477444fb-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:55.098332 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:55.098298    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kr7q\" (UniqueName: \"kubernetes.io/projected/4cacfc24-be5d-4505-a761-18aa477444fb-kube-api-access-6kr7q\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:55.098332 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:55.098308    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cacfc24-be5d-4505-a761-18aa477444fb-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:23:55.673682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:55.673650    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n"
Oct 30 01:23:55.673872 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:55.673682    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n" event={"ID":"4cacfc24-be5d-4505-a761-18aa477444fb","Type":"ContainerDied","Data":"e85734b8013230c1fe7a140fd5f11d5267319837be488bfca5edddd762fbf121"}
Oct 30 01:23:55.673872 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:55.673738    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e85734b8013230c1fe7a140fd5f11d5267319837be488bfca5edddd762fbf121"
Oct 30 01:23:58.654969 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:58.654940    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw"]
Oct 30 01:23:58.659674 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:58.659649    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-60e8f4-cchmw"]
Oct 30 01:23:59.558614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:59.558583    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx"]
Oct 30 01:23:59.559141 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:59.559127    6099 cpu_manager.go:414] "containerMap: removing container" podUID="4cacfc24-be5d-4505-a761-18aa477444fb" containerName="revoke-keys"
Oct 30 01:23:59.559187 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:59.559142    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cacfc24-be5d-4505-a761-18aa477444fb" containerName="revoke-keys"
Oct 30 01:23:59.559277 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:59.559266    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cacfc24-be5d-4505-a761-18aa477444fb" containerName="revoke-keys"
Oct 30 01:23:59.564331 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:59.564302    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx"
Oct 30 01:23:59.572248 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:59.572216    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxwdd\" (UniqueName: \"kubernetes.io/projected/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-kube-api-access-cxwdd\") pod \"maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx\" (UID: \"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx"
Oct 30 01:23:59.572616 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:59.572590    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx\" (UID: \"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx"
Oct 30 01:23:59.573169 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:59.572988    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx\" (UID: \"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx"
Oct 30 01:23:59.581903 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:59.581878    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx"]
Oct 30 01:23:59.614575 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:59.614543    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:23:59.614752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:59.614589    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:23:59.674337 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:59.674306    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx\" (UID: \"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx"
Oct 30 01:23:59.674735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:59.674439    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx\" (UID: \"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx"
Oct 30 01:23:59.674735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:59.674469    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxwdd\" (UniqueName: \"kubernetes.io/projected/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-kube-api-access-cxwdd\") pod \"maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx\" (UID: \"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx"
Oct 30 01:23:59.675073 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:59.675049    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx\" (UID: \"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx"
Oct 30 01:23:59.676846 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:59.676817    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx\" (UID: \"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx"
Oct 30 01:23:59.697608 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:59.697574    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxwdd\" (UniqueName: \"kubernetes.io/projected/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-kube-api-access-cxwdd\") pod \"maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx\" (UID: \"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx"
Oct 30 01:23:59.881840 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:23:59.881761    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx"
Oct 30 01:24:00.182454 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.182422    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx"]
Oct 30 01:24:00.182768 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:24:00.182742    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8771a3ad_b84a_4e90_9fab_53f1f8c6c14f.slice/crio-8824605bdbc86012f4ed3a60d31b7d9c5b3a8f83fe51c6e88f4549f7ae57b51d WatchSource:0}: Error finding container 8824605bdbc86012f4ed3a60d31b7d9c5b3a8f83fe51c6e88f4549f7ae57b51d: Status 404 returned error can't find the container with id 8824605bdbc86012f4ed3a60d31b7d9c5b3a8f83fe51c6e88f4549f7ae57b51d
Oct 30 01:24:00.261395 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.261364    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0" path="/var/lib/kubelet/pods/fafeae69-e8b5-49e7-8f3a-10ca12e3c7e0/volumes"
Oct 30 01:24:00.606077 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.606042    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main" containerID="cri-o://fcbefd856a76c823e0f9502b98fabf035e638089a38641cc1fbd6ba7da4f02e8" gracePeriod=45
Oct 30 01:24:00.615154 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.615111    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main" containerID="cri-o://3e72a155d8f701e715e3c9edabeb14424014731d3caffcf8415335d317b2c078" gracePeriod=45
Oct 30 01:24:00.710542 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.710504    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx" event={"ID":"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f","Type":"ContainerStarted","Data":"b41ba09aeb529035c0851101f185c098b93c07ef2d9784f7e87aabd3d4358101"}
Oct 30 01:24:00.710951 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.710563    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx" event={"ID":"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f","Type":"ContainerStarted","Data":"8824605bdbc86012f4ed3a60d31b7d9c5b3a8f83fe51c6e88f4549f7ae57b51d"}
Oct 30 01:24:00.753250 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.753191    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx" podStartSLOduration=1.753177084 podStartE2EDuration="1.753177084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:00.751012045 +0000 UTC m=+2934.607919800" watchObservedRunningTime="2026-10-30 01:24:00.753177084 +0000 UTC m=+2934.610084895"
Oct 30 01:24:00.880687 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.880654    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 01:24:00.912688 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.912665    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 01:24:00.988377 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.988292    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5daab803-f655-41c9-98d9-7049b1ba3597-kube-api-access-f8279\" (UniqueName: \"kubernetes.io/projected/5daab803-f655-41c9-98d9-7049b1ba3597-kube-api-access-f8279\") pod \"5daab803-f655-41c9-98d9-7049b1ba3597\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") "
Oct 30 01:24:00.988377 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.988335    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5daab803-f655-41c9-98d9-7049b1ba3597-tls-certs\" (UniqueName: \"kubernetes.io/secret/5daab803-f655-41c9-98d9-7049b1ba3597-tls-certs\") pod \"5daab803-f655-41c9-98d9-7049b1ba3597\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") "
Oct 30 01:24:00.988377 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.988377    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-dshm\") pod \"5daab803-f655-41c9-98d9-7049b1ba3597\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") "
Oct 30 01:24:00.988750 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.988413    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-model-cache\") pod \"24858435-0625-4512-bef6-9e9f3a3bb6e2\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") "
Oct 30 01:24:00.988750 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.988620    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-tmp-dir\") pod \"24858435-0625-4512-bef6-9e9f3a3bb6e2\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") "
Oct 30 01:24:00.988750 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.988684    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-model-cache" pod "24858435-0625-4512-bef6-9e9f3a3bb6e2" (UID: "24858435-0625-4512-bef6-9e9f3a3bb6e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:00.988951 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.988660    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24858435-0625-4512-bef6-9e9f3a3bb6e2-tls-certs\" (UniqueName: \"kubernetes.io/secret/24858435-0625-4512-bef6-9e9f3a3bb6e2-tls-certs\") pod \"24858435-0625-4512-bef6-9e9f3a3bb6e2\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") "
Oct 30 01:24:00.989023 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.988978    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-home\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-home\") pod \"5daab803-f655-41c9-98d9-7049b1ba3597\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") "
Oct 30 01:24:00.989132 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.989117    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-tmp-dir\") pod \"5daab803-f655-41c9-98d9-7049b1ba3597\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") "
Oct 30 01:24:00.989229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.989194    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-dshm\") pod \"24858435-0625-4512-bef6-9e9f3a3bb6e2\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") "
Oct 30 01:24:00.989335 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.989294    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-model-cache\") pod \"5daab803-f655-41c9-98d9-7049b1ba3597\" (UID: \"5daab803-f655-41c9-98d9-7049b1ba3597\") "
Oct 30 01:24:00.989423 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.989339    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24858435-0625-4512-bef6-9e9f3a3bb6e2-kube-api-access-tmckk\" (UniqueName: \"kubernetes.io/projected/24858435-0625-4512-bef6-9e9f3a3bb6e2-kube-api-access-tmckk\") pod \"24858435-0625-4512-bef6-9e9f3a3bb6e2\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") "
Oct 30 01:24:00.989423 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.989374    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-home\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-home\") pod \"24858435-0625-4512-bef6-9e9f3a3bb6e2\" (UID: \"24858435-0625-4512-bef6-9e9f3a3bb6e2\") "
Oct 30 01:24:00.991080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.990036    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-tmp-dir" pod "24858435-0625-4512-bef6-9e9f3a3bb6e2" (UID: "24858435-0625-4512-bef6-9e9f3a3bb6e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:00.991080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.990058    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-model-cache" pod "5daab803-f655-41c9-98d9-7049b1ba3597" (UID: "5daab803-f655-41c9-98d9-7049b1ba3597"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:00.991080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.990292    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-home" pod "5daab803-f655-41c9-98d9-7049b1ba3597" (UID: "5daab803-f655-41c9-98d9-7049b1ba3597"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:00.991080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.990487    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-tmp-dir" pod "5daab803-f655-41c9-98d9-7049b1ba3597" (UID: "5daab803-f655-41c9-98d9-7049b1ba3597"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:00.991080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.990629    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-home" pod "24858435-0625-4512-bef6-9e9f3a3bb6e2" (UID: "24858435-0625-4512-bef6-9e9f3a3bb6e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:00.991080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.990975    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:00.991080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.991001    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:00.991080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.991014    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:00.991080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.991028    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:00.991080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.991041    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:00.991080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.991053    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:00.991844 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.991395    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5daab803-f655-41c9-98d9-7049b1ba3597-tls-certs" pod "5daab803-f655-41c9-98d9-7049b1ba3597" (UID: "5daab803-f655-41c9-98d9-7049b1ba3597"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:00.992084 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.992040    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-dshm" pod "5daab803-f655-41c9-98d9-7049b1ba3597" (UID: "5daab803-f655-41c9-98d9-7049b1ba3597"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:00.992276 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.992253    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24858435-0625-4512-bef6-9e9f3a3bb6e2-tls-certs" pod "24858435-0625-4512-bef6-9e9f3a3bb6e2" (UID: "24858435-0625-4512-bef6-9e9f3a3bb6e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:00.992348 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.992272    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5daab803-f655-41c9-98d9-7049b1ba3597-kube-api-access-f8279" pod "5daab803-f655-41c9-98d9-7049b1ba3597" (UID: "5daab803-f655-41c9-98d9-7049b1ba3597"). InnerVolumeSpecName "kube-api-access-f8279". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:00.992809 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.992785    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24858435-0625-4512-bef6-9e9f3a3bb6e2-kube-api-access-tmckk" pod "24858435-0625-4512-bef6-9e9f3a3bb6e2" (UID: "24858435-0625-4512-bef6-9e9f3a3bb6e2"). InnerVolumeSpecName "kube-api-access-tmckk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:00.993345 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:00.993323    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-dshm" pod "24858435-0625-4512-bef6-9e9f3a3bb6e2" (UID: "24858435-0625-4512-bef6-9e9f3a3bb6e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:01.092047 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.092024    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5daab803-f655-41c9-98d9-7049b1ba3597-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:01.092047 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.092048    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24858435-0625-4512-bef6-9e9f3a3bb6e2-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:01.092194 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.092057    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24858435-0625-4512-bef6-9e9f3a3bb6e2-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:01.092194 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.092065    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmckk\" (UniqueName: \"kubernetes.io/projected/24858435-0625-4512-bef6-9e9f3a3bb6e2-kube-api-access-tmckk\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:01.092194 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.092075    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8279\" (UniqueName: \"kubernetes.io/projected/5daab803-f655-41c9-98d9-7049b1ba3597-kube-api-access-f8279\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:01.092194 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.092085    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5daab803-f655-41c9-98d9-7049b1ba3597-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:01.718078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.718043    6099 generic.go:356] "Generic (PLEG): container finished" podID="5daab803-f655-41c9-98d9-7049b1ba3597" containerID="fcbefd856a76c823e0f9502b98fabf035e638089a38641cc1fbd6ba7da4f02e8" exitCode=0
Oct 30 01:24:01.718559 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.718082    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"
Oct 30 01:24:01.718559 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.718129    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" event={"ID":"5daab803-f655-41c9-98d9-7049b1ba3597","Type":"ContainerDied","Data":"fcbefd856a76c823e0f9502b98fabf035e638089a38641cc1fbd6ba7da4f02e8"}
Oct 30 01:24:01.718559 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.718174    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx" event={"ID":"5daab803-f655-41c9-98d9-7049b1ba3597","Type":"ContainerDied","Data":"3cb9ba7639d608e779d36a1496acd498a56664b82f6726c9197ee3b912b947c4"}
Oct 30 01:24:01.718559 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.718190    6099 scope.go:122] "RemoveContainer" containerID="fcbefd856a76c823e0f9502b98fabf035e638089a38641cc1fbd6ba7da4f02e8"
Oct 30 01:24:01.720060 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.720038    6099 generic.go:356] "Generic (PLEG): container finished" podID="8771a3ad-b84a-4e90-9fab-53f1f8c6c14f" containerID="b41ba09aeb529035c0851101f185c098b93c07ef2d9784f7e87aabd3d4358101" exitCode=0
Oct 30 01:24:01.720198 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.720086    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx" event={"ID":"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f","Type":"ContainerDied","Data":"b41ba09aeb529035c0851101f185c098b93c07ef2d9784f7e87aabd3d4358101"}
Oct 30 01:24:01.721591 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.721567    6099 generic.go:356] "Generic (PLEG): container finished" podID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerID="3e72a155d8f701e715e3c9edabeb14424014731d3caffcf8415335d317b2c078" exitCode=0
Oct 30 01:24:01.721822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.721613    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"
Oct 30 01:24:01.721822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.721647    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" event={"ID":"24858435-0625-4512-bef6-9e9f3a3bb6e2","Type":"ContainerDied","Data":"3e72a155d8f701e715e3c9edabeb14424014731d3caffcf8415335d317b2c078"}
Oct 30 01:24:01.721822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.721675    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld" event={"ID":"24858435-0625-4512-bef6-9e9f3a3bb6e2","Type":"ContainerDied","Data":"928bfd756666cae39f561f7b67741e2ff3ac7ff396d6aff40ed9318782abf5ac"}
Oct 30 01:24:01.746950 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.746223    6099 scope.go:122] "RemoveContainer" containerID="fcbefd856a76c823e0f9502b98fabf035e638089a38641cc1fbd6ba7da4f02e8"
Oct 30 01:24:01.747123 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:24:01.747094    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcbefd856a76c823e0f9502b98fabf035e638089a38641cc1fbd6ba7da4f02e8\": container with ID starting with fcbefd856a76c823e0f9502b98fabf035e638089a38641cc1fbd6ba7da4f02e8 not found: ID does not exist" containerID="fcbefd856a76c823e0f9502b98fabf035e638089a38641cc1fbd6ba7da4f02e8"
Oct 30 01:24:01.747222 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.747133    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcbefd856a76c823e0f9502b98fabf035e638089a38641cc1fbd6ba7da4f02e8"} err="failed to get container status \"fcbefd856a76c823e0f9502b98fabf035e638089a38641cc1fbd6ba7da4f02e8\": rpc error: code = NotFound desc = could not find container \"fcbefd856a76c823e0f9502b98fabf035e638089a38641cc1fbd6ba7da4f02e8\": container with ID starting with fcbefd856a76c823e0f9502b98fabf035e638089a38641cc1fbd6ba7da4f02e8 not found: ID does not exist"
Oct 30 01:24:01.747222 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.747158    6099 scope.go:122] "RemoveContainer" containerID="3e72a155d8f701e715e3c9edabeb14424014731d3caffcf8415335d317b2c078"
Oct 30 01:24:01.757692 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.757673    6099 scope.go:122] "RemoveContainer" containerID="3e72a155d8f701e715e3c9edabeb14424014731d3caffcf8415335d317b2c078"
Oct 30 01:24:01.757967 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:24:01.757951    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e72a155d8f701e715e3c9edabeb14424014731d3caffcf8415335d317b2c078\": container with ID starting with 3e72a155d8f701e715e3c9edabeb14424014731d3caffcf8415335d317b2c078 not found: ID does not exist" containerID="3e72a155d8f701e715e3c9edabeb14424014731d3caffcf8415335d317b2c078"
Oct 30 01:24:01.758011 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.757973    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e72a155d8f701e715e3c9edabeb14424014731d3caffcf8415335d317b2c078"} err="failed to get container status \"3e72a155d8f701e715e3c9edabeb14424014731d3caffcf8415335d317b2c078\": rpc error: code = NotFound desc = could not find container \"3e72a155d8f701e715e3c9edabeb14424014731d3caffcf8415335d317b2c078\": container with ID starting with 3e72a155d8f701e715e3c9edabeb14424014731d3caffcf8415335d317b2c078 not found: ID does not exist"
Oct 30 01:24:01.827739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.827696    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"]
Oct 30 01:24:01.831252 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.831227    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/distinct-w2-40ac41-kserve-5c9c4f8bd4-wh5bx"]
Oct 30 01:24:01.864387 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.864358    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"]
Oct 30 01:24:01.870985 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:01.870961    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-40ac41/premium-w2-40ac41-kserve-7bdcdb48df-hjpld"]
Oct 30 01:24:02.262240 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:02.262210    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" path="/var/lib/kubelet/pods/24858435-0625-4512-bef6-9e9f3a3bb6e2/volumes"
Oct 30 01:24:02.262749 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:02.262734    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" path="/var/lib/kubelet/pods/5daab803-f655-41c9-98d9-7049b1ba3597/volumes"
Oct 30 01:24:02.887418 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:02.887393    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx"
Oct 30 01:24:02.913970 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:02.913943    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-maas-api-xks-ca\") pod \"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f\" (UID: \"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f\") "
Oct 30 01:24:02.914114 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:02.913984    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-maas-api-openshift-ca\") pod \"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f\" (UID: \"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f\") "
Oct 30 01:24:02.914114 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:02.914013    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-kube-api-access-cxwdd\" (UniqueName: \"kubernetes.io/projected/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-kube-api-access-cxwdd\") pod \"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f\" (UID: \"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f\") "
Oct 30 01:24:02.914395 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:02.914367    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-maas-api-openshift-ca" pod "8771a3ad-b84a-4e90-9fab-53f1f8c6c14f" (UID: "8771a3ad-b84a-4e90-9fab-53f1f8c6c14f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:02.916064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:02.916042    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-maas-api-xks-ca" pod "8771a3ad-b84a-4e90-9fab-53f1f8c6c14f" (UID: "8771a3ad-b84a-4e90-9fab-53f1f8c6c14f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:02.916196 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:02.916177    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-kube-api-access-cxwdd" pod "8771a3ad-b84a-4e90-9fab-53f1f8c6c14f" (UID: "8771a3ad-b84a-4e90-9fab-53f1f8c6c14f"). InnerVolumeSpecName "kube-api-access-cxwdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:03.016078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:03.015997    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.016078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:03.016026    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.016078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:03.016036    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxwdd\" (UniqueName: \"kubernetes.io/projected/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f-kube-api-access-cxwdd\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.739535 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:03.739509    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx"
Oct 30 01:24:03.739783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:03.739505    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx" event={"ID":"8771a3ad-b84a-4e90-9fab-53f1f8c6c14f","Type":"ContainerDied","Data":"8824605bdbc86012f4ed3a60d31b7d9c5b3a8f83fe51c6e88f4549f7ae57b51d"}
Oct 30 01:24:03.739783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:03.739650    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8824605bdbc86012f4ed3a60d31b7d9c5b3a8f83fe51c6e88f4549f7ae57b51d"
Oct 30 01:24:04.737164 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:04.737129    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95" podUID="41feca8d-0a29-42e4-9969-c1f6e4ee7f36" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:24:05.502492 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:05.502458    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"]
Oct 30 01:24:05.502752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:05.502718    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc" podUID="b9004b35-ce7c-45b6-a0d0-94a16203ece2" containerName="istio-proxy" containerID="cri-o://d4304c788ceea067ebdfd5204554a2a9388b6cfab8db76af130bbecb00c45b69" gracePeriod=30
Oct 30 01:24:06.153658 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.153622    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8"]
Oct 30 01:24:06.154254 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.154237    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main"
Oct 30 01:24:06.154300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.154255    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main"
Oct 30 01:24:06.154300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.154281    6099 cpu_manager.go:414] "containerMap: removing container" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main"
Oct 30 01:24:06.154300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.154286    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main"
Oct 30 01:24:06.154300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.154300    6099 cpu_manager.go:414] "containerMap: removing container" podUID="8771a3ad-b84a-4e90-9fab-53f1f8c6c14f" containerName="revoke-keys"
Oct 30 01:24:06.154447 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.154306    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8771a3ad-b84a-4e90-9fab-53f1f8c6c14f" containerName="revoke-keys"
Oct 30 01:24:06.154447 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.154415    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5daab803-f655-41c9-98d9-7049b1ba3597" containerName="main"
Oct 30 01:24:06.154447 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.154428    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="24858435-0625-4512-bef6-9e9f3a3bb6e2" containerName="main"
Oct 30 01:24:06.154447 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.154436    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="8771a3ad-b84a-4e90-9fab-53f1f8c6c14f" containerName="revoke-keys"
Oct 30 01:24:06.158099 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.158083    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8"
Oct 30 01:24:06.177739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.177715    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8"]
Oct 30 01:24:06.190755 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.190697    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:24:06.190911 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.190850    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:24:06.255865 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.255833    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3f9333-df3a-4838-801e-b45201f33f25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8\" (UID: \"dd3f9333-df3a-4838-801e-b45201f33f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8"
Oct 30 01:24:06.256043 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.256021    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3f9333-df3a-4838-801e-b45201f33f25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8\" (UID: \"dd3f9333-df3a-4838-801e-b45201f33f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8"
Oct 30 01:24:06.256106 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.256093    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvcfd\" (UniqueName: \"kubernetes.io/projected/dd3f9333-df3a-4838-801e-b45201f33f25-kube-api-access-dvcfd\") pod \"maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8\" (UID: \"dd3f9333-df3a-4838-801e-b45201f33f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8"
Oct 30 01:24:06.357496 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.357454    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3f9333-df3a-4838-801e-b45201f33f25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8\" (UID: \"dd3f9333-df3a-4838-801e-b45201f33f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8"
Oct 30 01:24:06.357686 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.357575    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3f9333-df3a-4838-801e-b45201f33f25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8\" (UID: \"dd3f9333-df3a-4838-801e-b45201f33f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8"
Oct 30 01:24:06.357686 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.357623    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvcfd\" (UniqueName: \"kubernetes.io/projected/dd3f9333-df3a-4838-801e-b45201f33f25-kube-api-access-dvcfd\") pod \"maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8\" (UID: \"dd3f9333-df3a-4838-801e-b45201f33f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8"
Oct 30 01:24:06.358268 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.358240    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3f9333-df3a-4838-801e-b45201f33f25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8\" (UID: \"dd3f9333-df3a-4838-801e-b45201f33f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8"
Oct 30 01:24:06.359813 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.359789    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3f9333-df3a-4838-801e-b45201f33f25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8\" (UID: \"dd3f9333-df3a-4838-801e-b45201f33f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8"
Oct 30 01:24:06.380733 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.380689    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvcfd\" (UniqueName: \"kubernetes.io/projected/dd3f9333-df3a-4838-801e-b45201f33f25-kube-api-access-dvcfd\") pod \"maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8\" (UID: \"dd3f9333-df3a-4838-801e-b45201f33f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8"
Oct 30 01:24:06.471978 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.471906    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8"
Oct 30 01:24:06.787389 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:06.787356    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8"]
Oct 30 01:24:06.788443 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:24:06.788414    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3f9333_df3a_4838_801e_b45201f33f25.slice/crio-146f37328647f77da3252a388c489dc988d1ad82143ff89a58ec2a7a3763c722 WatchSource:0}: Error finding container 146f37328647f77da3252a388c489dc988d1ad82143ff89a58ec2a7a3763c722: Status 404 returned error can't find the container with id 146f37328647f77da3252a388c489dc988d1ad82143ff89a58ec2a7a3763c722
Oct 30 01:24:07.768992 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:07.768952    6099 generic.go:356] "Generic (PLEG): container finished" podID="dd3f9333-df3a-4838-801e-b45201f33f25" containerID="7abbefdaafb1c394cb16963f5f0b9afe921185cc83a82daddb121978af1961c7" exitCode=0
Oct 30 01:24:07.769402 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:07.769038    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8" event={"ID":"dd3f9333-df3a-4838-801e-b45201f33f25","Type":"ContainerDied","Data":"7abbefdaafb1c394cb16963f5f0b9afe921185cc83a82daddb121978af1961c7"}
Oct 30 01:24:07.769402 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:07.769121    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8" event={"ID":"dd3f9333-df3a-4838-801e-b45201f33f25","Type":"ContainerStarted","Data":"146f37328647f77da3252a388c489dc988d1ad82143ff89a58ec2a7a3763c722"}
Oct 30 01:24:08.577819 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:08.577761    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95"
Oct 30 01:24:08.688273 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:08.688243    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41feca8d-0a29-42e4-9969-c1f6e4ee7f36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41feca8d-0a29-42e4-9969-c1f6e4ee7f36-plugins-config-volume\") pod \"41feca8d-0a29-42e4-9969-c1f6e4ee7f36\" (UID: \"41feca8d-0a29-42e4-9969-c1f6e4ee7f36\") "
Oct 30 01:24:08.688446 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:08.688312    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41feca8d-0a29-42e4-9969-c1f6e4ee7f36-kube-api-access-n689r\" (UniqueName: \"kubernetes.io/projected/41feca8d-0a29-42e4-9969-c1f6e4ee7f36-kube-api-access-n689r\") pod \"41feca8d-0a29-42e4-9969-c1f6e4ee7f36\" (UID: \"41feca8d-0a29-42e4-9969-c1f6e4ee7f36\") "
Oct 30 01:24:08.688685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:08.688661    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41feca8d-0a29-42e4-9969-c1f6e4ee7f36-plugins-config-volume" pod "41feca8d-0a29-42e4-9969-c1f6e4ee7f36" (UID: "41feca8d-0a29-42e4-9969-c1f6e4ee7f36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:08.688992 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:08.688960    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41feca8d-0a29-42e4-9969-c1f6e4ee7f36-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:08.690458 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:08.690430    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41feca8d-0a29-42e4-9969-c1f6e4ee7f36-kube-api-access-n689r" pod "41feca8d-0a29-42e4-9969-c1f6e4ee7f36" (UID: "41feca8d-0a29-42e4-9969-c1f6e4ee7f36"). InnerVolumeSpecName "kube-api-access-n689r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:08.777146 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:08.777115    6099 generic.go:356] "Generic (PLEG): container finished" podID="41feca8d-0a29-42e4-9969-c1f6e4ee7f36" containerID="9f63313577e881b892d25c7b079c4f25dec08298c35f2a6750860351198f2eda" exitCode=0
Oct 30 01:24:08.777501 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:08.777163    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95"
Oct 30 01:24:08.777501 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:08.777196    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95" event={"ID":"41feca8d-0a29-42e4-9969-c1f6e4ee7f36","Type":"ContainerDied","Data":"9f63313577e881b892d25c7b079c4f25dec08298c35f2a6750860351198f2eda"}
Oct 30 01:24:08.777501 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:08.777249    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95" event={"ID":"41feca8d-0a29-42e4-9969-c1f6e4ee7f36","Type":"ContainerDied","Data":"99cfff83f6db7556d932c2fc873483b1b685878149398109fb5f2d2584f7570d"}
Oct 30 01:24:08.777501 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:08.777275    6099 scope.go:122] "RemoveContainer" containerID="9f63313577e881b892d25c7b079c4f25dec08298c35f2a6750860351198f2eda"
Oct 30 01:24:08.788835 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:08.788818    6099 scope.go:122] "RemoveContainer" containerID="9f63313577e881b892d25c7b079c4f25dec08298c35f2a6750860351198f2eda"
Oct 30 01:24:08.789129 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:24:08.789106    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f63313577e881b892d25c7b079c4f25dec08298c35f2a6750860351198f2eda\": container with ID starting with 9f63313577e881b892d25c7b079c4f25dec08298c35f2a6750860351198f2eda not found: ID does not exist" containerID="9f63313577e881b892d25c7b079c4f25dec08298c35f2a6750860351198f2eda"
Oct 30 01:24:08.789237 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:08.789134    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f63313577e881b892d25c7b079c4f25dec08298c35f2a6750860351198f2eda"} err="failed to get container status \"9f63313577e881b892d25c7b079c4f25dec08298c35f2a6750860351198f2eda\": rpc error: code = NotFound desc = could not find container \"9f63313577e881b892d25c7b079c4f25dec08298c35f2a6750860351198f2eda\": container with ID starting with 9f63313577e881b892d25c7b079c4f25dec08298c35f2a6750860351198f2eda not found: ID does not exist"
Oct 30 01:24:08.789665 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:08.789647    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n689r\" (UniqueName: \"kubernetes.io/projected/41feca8d-0a29-42e4-9969-c1f6e4ee7f36-kube-api-access-n689r\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:08.830337 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:08.830234    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95"]
Oct 30 01:24:08.834739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:08.834696    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-60e8f4-6b8d55bf6c-wcc95"]
Oct 30 01:24:08.954038 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:08.954013    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8"
Oct 30 01:24:09.093564 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:09.093487    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd3f9333-df3a-4838-801e-b45201f33f25-kube-api-access-dvcfd\" (UniqueName: \"kubernetes.io/projected/dd3f9333-df3a-4838-801e-b45201f33f25-kube-api-access-dvcfd\") pod \"dd3f9333-df3a-4838-801e-b45201f33f25\" (UID: \"dd3f9333-df3a-4838-801e-b45201f33f25\") "
Oct 30 01:24:09.093564 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:09.093552    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd3f9333-df3a-4838-801e-b45201f33f25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3f9333-df3a-4838-801e-b45201f33f25-maas-api-xks-ca\") pod \"dd3f9333-df3a-4838-801e-b45201f33f25\" (UID: \"dd3f9333-df3a-4838-801e-b45201f33f25\") "
Oct 30 01:24:09.093856 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:09.093691    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3f9333-df3a-4838-801e-b45201f33f25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3f9333-df3a-4838-801e-b45201f33f25-maas-api-openshift-ca\") pod \"dd3f9333-df3a-4838-801e-b45201f33f25\" (UID: \"dd3f9333-df3a-4838-801e-b45201f33f25\") "
Oct 30 01:24:09.094078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:09.094055    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3f9333-df3a-4838-801e-b45201f33f25-maas-api-openshift-ca" pod "dd3f9333-df3a-4838-801e-b45201f33f25" (UID: "dd3f9333-df3a-4838-801e-b45201f33f25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:09.094162 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:09.094131    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3f9333-df3a-4838-801e-b45201f33f25-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:09.095615 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:09.095591    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3f9333-df3a-4838-801e-b45201f33f25-maas-api-xks-ca" pod "dd3f9333-df3a-4838-801e-b45201f33f25" (UID: "dd3f9333-df3a-4838-801e-b45201f33f25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:09.095783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:09.095647    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3f9333-df3a-4838-801e-b45201f33f25-kube-api-access-dvcfd" pod "dd3f9333-df3a-4838-801e-b45201f33f25" (UID: "dd3f9333-df3a-4838-801e-b45201f33f25"). InnerVolumeSpecName "kube-api-access-dvcfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:09.194892 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:09.194867    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvcfd\" (UniqueName: \"kubernetes.io/projected/dd3f9333-df3a-4838-801e-b45201f33f25-kube-api-access-dvcfd\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:09.194892 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:09.194894    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3f9333-df3a-4838-801e-b45201f33f25-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:09.787464 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:09.787429    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8" event={"ID":"dd3f9333-df3a-4838-801e-b45201f33f25","Type":"ContainerDied","Data":"146f37328647f77da3252a388c489dc988d1ad82143ff89a58ec2a7a3763c722"}
Oct 30 01:24:09.787464 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:09.787444    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8"
Oct 30 01:24:09.788043 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:09.787475    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="146f37328647f77da3252a388c489dc988d1ad82143ff89a58ec2a7a3763c722"
Oct 30 01:24:10.261964 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.261935    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41feca8d-0a29-42e4-9969-c1f6e4ee7f36" path="/var/lib/kubelet/pods/41feca8d-0a29-42e4-9969-c1f6e4ee7f36/volumes"
Oct 30 01:24:10.778370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.778309    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:24:10.795480 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.795449    6099 generic.go:356] "Generic (PLEG): container finished" podID="b9004b35-ce7c-45b6-a0d0-94a16203ece2" containerID="d4304c788ceea067ebdfd5204554a2a9388b6cfab8db76af130bbecb00c45b69" exitCode=0
Oct 30 01:24:10.795918 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.795492    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"
Oct 30 01:24:10.795918 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.795530    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc" event={"ID":"b9004b35-ce7c-45b6-a0d0-94a16203ece2","Type":"ContainerDied","Data":"d4304c788ceea067ebdfd5204554a2a9388b6cfab8db76af130bbecb00c45b69"}
Oct 30 01:24:10.795918 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.795580    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc" event={"ID":"b9004b35-ce7c-45b6-a0d0-94a16203ece2","Type":"ContainerDied","Data":"07b8b408b4b6d9d29049a37cd1650b94175263c8cb38f4eccf076ca13b852182"}
Oct 30 01:24:10.795918 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.795602    6099 scope.go:122] "RemoveContainer" containerID="d4304c788ceea067ebdfd5204554a2a9388b6cfab8db76af130bbecb00c45b69"
Oct 30 01:24:10.805546 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.805531    6099 scope.go:122] "RemoveContainer" containerID="d4304c788ceea067ebdfd5204554a2a9388b6cfab8db76af130bbecb00c45b69"
Oct 30 01:24:10.805780 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:24:10.805757    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4304c788ceea067ebdfd5204554a2a9388b6cfab8db76af130bbecb00c45b69\": container with ID starting with d4304c788ceea067ebdfd5204554a2a9388b6cfab8db76af130bbecb00c45b69 not found: ID does not exist" containerID="d4304c788ceea067ebdfd5204554a2a9388b6cfab8db76af130bbecb00c45b69"
Oct 30 01:24:10.805849 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.805789    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4304c788ceea067ebdfd5204554a2a9388b6cfab8db76af130bbecb00c45b69"} err="failed to get container status \"d4304c788ceea067ebdfd5204554a2a9388b6cfab8db76af130bbecb00c45b69\": rpc error: code = NotFound desc = could not find container \"d4304c788ceea067ebdfd5204554a2a9388b6cfab8db76af130bbecb00c45b69\": container with ID starting with d4304c788ceea067ebdfd5204554a2a9388b6cfab8db76af130bbecb00c45b69 not found: ID does not exist"
Oct 30 01:24:10.914285 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.914252    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-podinfo\") pod \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") "
Oct 30 01:24:10.914450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.914306    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-data\") pod \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") "
Oct 30 01:24:10.914450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.914334    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-workload-certs\") pod \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") "
Oct 30 01:24:10.914450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.914429    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istiod-ca-cert\") pod \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") "
Oct 30 01:24:10.914607 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.914462    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9004b35-ce7c-45b6-a0d0-94a16203ece2-kube-api-access-t6r8m\" (UniqueName: \"kubernetes.io/projected/b9004b35-ce7c-45b6-a0d0-94a16203ece2-kube-api-access-t6r8m\") pod \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") "
Oct 30 01:24:10.914607 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.914485    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-workload-socket\") pod \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") "
Oct 30 01:24:10.914607 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.914529    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-credential-socket\") pod \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") "
Oct 30 01:24:10.914813 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.914631    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-envoy\") pod \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") "
Oct 30 01:24:10.914813 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.914681    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-token\" (UniqueName: \"kubernetes.io/projected/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-token\") pod \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\" (UID: \"b9004b35-ce7c-45b6-a0d0-94a16203ece2\") "
Oct 30 01:24:10.914813 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.914690    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-data" pod "b9004b35-ce7c-45b6-a0d0-94a16203ece2" (UID: "b9004b35-ce7c-45b6-a0d0-94a16203ece2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:10.914813 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.914763    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-workload-certs" pod "b9004b35-ce7c-45b6-a0d0-94a16203ece2" (UID: "b9004b35-ce7c-45b6-a0d0-94a16203ece2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:10.915042 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.914927    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istiod-ca-cert" pod "b9004b35-ce7c-45b6-a0d0-94a16203ece2" (UID: "b9004b35-ce7c-45b6-a0d0-94a16203ece2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:10.915107 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.915056    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-credential-socket" pod "b9004b35-ce7c-45b6-a0d0-94a16203ece2" (UID: "b9004b35-ce7c-45b6-a0d0-94a16203ece2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:10.915275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.915236    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-workload-socket" pod "b9004b35-ce7c-45b6-a0d0-94a16203ece2" (UID: "b9004b35-ce7c-45b6-a0d0-94a16203ece2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:10.915429 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.915411    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:10.915429 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.915428    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:10.915617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.915438    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:10.915617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.915446    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:10.915617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.915454    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:10.916731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.916687    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-envoy" pod "b9004b35-ce7c-45b6-a0d0-94a16203ece2" (UID: "b9004b35-ce7c-45b6-a0d0-94a16203ece2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:10.916826 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.916795    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-podinfo" pod "b9004b35-ce7c-45b6-a0d0-94a16203ece2" (UID: "b9004b35-ce7c-45b6-a0d0-94a16203ece2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:24:10.917119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.917102    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-token" pod "b9004b35-ce7c-45b6-a0d0-94a16203ece2" (UID: "b9004b35-ce7c-45b6-a0d0-94a16203ece2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:10.917189 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.917127    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9004b35-ce7c-45b6-a0d0-94a16203ece2-kube-api-access-t6r8m" pod "b9004b35-ce7c-45b6-a0d0-94a16203ece2" (UID: "b9004b35-ce7c-45b6-a0d0-94a16203ece2"). InnerVolumeSpecName "kube-api-access-t6r8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:10.943035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.943009    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"]
Oct 30 01:24:10.948146 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.948123    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"]
Oct 30 01:24:10.955617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.955594    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"]
Oct 30 01:24:10.963348 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:10.963310    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"]
Oct 30 01:24:11.016488 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:11.016458    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6r8m\" (UniqueName: \"kubernetes.io/projected/b9004b35-ce7c-45b6-a0d0-94a16203ece2-kube-api-access-t6r8m\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.016488 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:11.016484    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.016488 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:11.016496    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.016745 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:11.016504    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9004b35-ce7c-45b6-a0d0-94a16203ece2-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.159670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:11.159639    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"]
Oct 30 01:24:11.162167 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:11.162143    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-60e8f4-openshift-default-5fffb9894c-dtkfc"]
Oct 30 01:24:12.262006 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:12.261963    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9004b35-ce7c-45b6-a0d0-94a16203ece2" path="/var/lib/kubelet/pods/b9004b35-ce7c-45b6-a0d0-94a16203ece2/volumes"
Oct 30 01:24:15.166467 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:15.166429    6099 scope.go:122] "RemoveContainer" containerID="96c5b021e92d1ef9159f2cad45b6085bbd0fe0bc3277f456b8f35733b29fb1d6"
Oct 30 01:24:15.837009 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:15.836967    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx"]
Oct 30 01:24:15.848344 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:15.848317    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5f99e7-w7ggx"]
Oct 30 01:24:16.262375 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:16.262293    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8771a3ad-b84a-4e90-9fab-53f1f8c6c14f" path="/var/lib/kubelet/pods/8771a3ad-b84a-4e90-9fab-53f1f8c6c14f/volumes"
Oct 30 01:24:16.633678 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:24:16.633642    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:24:18.072355 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:18.072321    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l"]
Oct 30 01:24:18.083990 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:18.083959    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-40ac41-gqb5l"]
Oct 30 01:24:18.260877 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:18.260842    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbf472f-6404-4020-8925-3413535f8207" path="/var/lib/kubelet/pods/bbbf472f-6404-4020-8925-3413535f8207/volumes"
Oct 30 01:24:24.737004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.736960    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht"]
Oct 30 01:24:24.737521 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.737507    6099 cpu_manager.go:414] "containerMap: removing container" podUID="dd3f9333-df3a-4838-801e-b45201f33f25" containerName="revoke-keys"
Oct 30 01:24:24.737579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.737522    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3f9333-df3a-4838-801e-b45201f33f25" containerName="revoke-keys"
Oct 30 01:24:24.737579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.737542    6099 cpu_manager.go:414] "containerMap: removing container" podUID="41feca8d-0a29-42e4-9969-c1f6e4ee7f36" containerName="payload-pre-processing"
Oct 30 01:24:24.737579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.737548    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41feca8d-0a29-42e4-9969-c1f6e4ee7f36" containerName="payload-pre-processing"
Oct 30 01:24:24.737579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.737562    6099 cpu_manager.go:414] "containerMap: removing container" podUID="b9004b35-ce7c-45b6-a0d0-94a16203ece2" containerName="istio-proxy"
Oct 30 01:24:24.737579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.737567    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9004b35-ce7c-45b6-a0d0-94a16203ece2" containerName="istio-proxy"
Oct 30 01:24:24.737774 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.737650    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3f9333-df3a-4838-801e-b45201f33f25" containerName="revoke-keys"
Oct 30 01:24:24.737774 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.737660    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="41feca8d-0a29-42e4-9969-c1f6e4ee7f36" containerName="payload-pre-processing"
Oct 30 01:24:24.737774 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.737669    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9004b35-ce7c-45b6-a0d0-94a16203ece2" containerName="istio-proxy"
Oct 30 01:24:24.743252 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.743232    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht"
Oct 30 01:24:24.803504 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.803464    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj7nn\" (UniqueName: \"kubernetes.io/projected/6098fd9c-5bb6-471b-8566-f845870c93c8-kube-api-access-wj7nn\") pod \"maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht\" (UID: \"6098fd9c-5bb6-471b-8566-f845870c93c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht"
Oct 30 01:24:24.803696 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.803614    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6098fd9c-5bb6-471b-8566-f845870c93c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht\" (UID: \"6098fd9c-5bb6-471b-8566-f845870c93c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht"
Oct 30 01:24:24.803696 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.803689    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6098fd9c-5bb6-471b-8566-f845870c93c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht\" (UID: \"6098fd9c-5bb6-471b-8566-f845870c93c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht"
Oct 30 01:24:24.825974 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.823887    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:24:24.825974 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.823918    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:24:24.826836 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.826811    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht"]
Oct 30 01:24:24.904549 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.904515    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6098fd9c-5bb6-471b-8566-f845870c93c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht\" (UID: \"6098fd9c-5bb6-471b-8566-f845870c93c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht"
Oct 30 01:24:24.904776 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.904607    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj7nn\" (UniqueName: \"kubernetes.io/projected/6098fd9c-5bb6-471b-8566-f845870c93c8-kube-api-access-wj7nn\") pod \"maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht\" (UID: \"6098fd9c-5bb6-471b-8566-f845870c93c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht"
Oct 30 01:24:24.904776 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.904757    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6098fd9c-5bb6-471b-8566-f845870c93c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht\" (UID: \"6098fd9c-5bb6-471b-8566-f845870c93c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht"
Oct 30 01:24:24.905542 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.905515    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6098fd9c-5bb6-471b-8566-f845870c93c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht\" (UID: \"6098fd9c-5bb6-471b-8566-f845870c93c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht"
Oct 30 01:24:24.907070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.907048    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6098fd9c-5bb6-471b-8566-f845870c93c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht\" (UID: \"6098fd9c-5bb6-471b-8566-f845870c93c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht"
Oct 30 01:24:24.934084 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:24.934051    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj7nn\" (UniqueName: \"kubernetes.io/projected/6098fd9c-5bb6-471b-8566-f845870c93c8-kube-api-access-wj7nn\") pod \"maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht\" (UID: \"6098fd9c-5bb6-471b-8566-f845870c93c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht"
Oct 30 01:24:25.059238 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:25.059205    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht"
Oct 30 01:24:25.366960 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:25.366925    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht"]
Oct 30 01:24:25.367952 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:24:25.367921    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6098fd9c_5bb6_471b_8566_f845870c93c8.slice/crio-d7313d2bcb12447aaa8c1edf0b48be5989f97ccdebc1513f8445bd6cbdeacf06 WatchSource:0}: Error finding container d7313d2bcb12447aaa8c1edf0b48be5989f97ccdebc1513f8445bd6cbdeacf06: Status 404 returned error can't find the container with id d7313d2bcb12447aaa8c1edf0b48be5989f97ccdebc1513f8445bd6cbdeacf06
Oct 30 01:24:25.902234 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:25.902200    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht" event={"ID":"6098fd9c-5bb6-471b-8566-f845870c93c8","Type":"ContainerStarted","Data":"ccb845bf03ebdf4918654d6745532b1d1aa0006c95dfa11a7ebdca4bbb17d39a"}
Oct 30 01:24:25.902636 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:25.902249    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht" event={"ID":"6098fd9c-5bb6-471b-8566-f845870c93c8","Type":"ContainerStarted","Data":"d7313d2bcb12447aaa8c1edf0b48be5989f97ccdebc1513f8445bd6cbdeacf06"}
Oct 30 01:24:25.958272 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:25.958167    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht" podStartSLOduration=1.958153998 podStartE2EDuration="1.958153998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:25.955901472 +0000 UTC m=+2959.812809228" watchObservedRunningTime="2026-10-30 01:24:25.958153998 +0000 UTC m=+2959.815061753"
Oct 30 01:24:25.965312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:25.965238    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main" containerID="cri-o://0f6befec8563c3fe8f865e5889831cf8c4558ba8addff016128fecd0772d8e0c" gracePeriod=45
Oct 30 01:24:25.971596 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:25.971559    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" containerID="cri-o://3a980ce7dee14d9664cf0b8ecf5377a3ea3f8c7b7204da49d9126599e400b2fe" gracePeriod=45
Oct 30 01:24:25.981249 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:25.981190    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main" containerID="cri-o://69980549eeb11bd271e6b6b7cbcf94319e08287456d151e919132bb6a3e6cd65" gracePeriod=45
Oct 30 01:24:25.983576 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:25.983546    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main" containerID="cri-o://d30d5c9bd2d3de754c957ed0667336034a8bfa55620acd2e9f34ccebae69b3b0" gracePeriod=45
Oct 30 01:24:26.187264 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.187109    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": dial tcp 10.134.0.91:8000: connect: connection refused"
Oct 30 01:24:26.274230 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.274201    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 01:24:26.323294 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.323264    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-home\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-home\") pod \"2967eaae-3210-4602-8b3f-f64ff80919cd\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") "
Oct 30 01:24:26.323434 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.323324    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-model-cache\") pod \"2967eaae-3210-4602-8b3f-f64ff80919cd\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") "
Oct 30 01:24:26.323434 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.323377    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-tmp-dir\") pod \"2967eaae-3210-4602-8b3f-f64ff80919cd\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") "
Oct 30 01:24:26.323523 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.323472    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2967eaae-3210-4602-8b3f-f64ff80919cd-kube-api-access-k6flc\" (UniqueName: \"kubernetes.io/projected/2967eaae-3210-4602-8b3f-f64ff80919cd-kube-api-access-k6flc\") pod \"2967eaae-3210-4602-8b3f-f64ff80919cd\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") "
Oct 30 01:24:26.323597 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.323573    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-home" pod "2967eaae-3210-4602-8b3f-f64ff80919cd" (UID: "2967eaae-3210-4602-8b3f-f64ff80919cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:26.323651 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.323597    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-tmp-dir" pod "2967eaae-3210-4602-8b3f-f64ff80919cd" (UID: "2967eaae-3210-4602-8b3f-f64ff80919cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:26.323651 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.323584    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-model-cache" pod "2967eaae-3210-4602-8b3f-f64ff80919cd" (UID: "2967eaae-3210-4602-8b3f-f64ff80919cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:26.323651 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.323643    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-dshm\") pod \"2967eaae-3210-4602-8b3f-f64ff80919cd\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") "
Oct 30 01:24:26.323784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.323668    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2967eaae-3210-4602-8b3f-f64ff80919cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/2967eaae-3210-4602-8b3f-f64ff80919cd-tls-certs\") pod \"2967eaae-3210-4602-8b3f-f64ff80919cd\" (UID: \"2967eaae-3210-4602-8b3f-f64ff80919cd\") "
Oct 30 01:24:26.324178 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.324155    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.324323 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.324181    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.324323 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.324195    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.327063 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.326384    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2967eaae-3210-4602-8b3f-f64ff80919cd-tls-certs" pod "2967eaae-3210-4602-8b3f-f64ff80919cd" (UID: "2967eaae-3210-4602-8b3f-f64ff80919cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:26.329146 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.329106    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2967eaae-3210-4602-8b3f-f64ff80919cd-kube-api-access-k6flc" pod "2967eaae-3210-4602-8b3f-f64ff80919cd" (UID: "2967eaae-3210-4602-8b3f-f64ff80919cd"). InnerVolumeSpecName "kube-api-access-k6flc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:26.329680 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.329656    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-dshm" pod "2967eaae-3210-4602-8b3f-f64ff80919cd" (UID: "2967eaae-3210-4602-8b3f-f64ff80919cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:26.351954 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.351928    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 01:24:26.355598 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.355570    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 01:24:26.359956 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.359933    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 01:24:26.425218 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425173    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-home\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-home\") pod \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") "
Oct 30 01:24:26.425218 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425215    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-home\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-home\") pod \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") "
Oct 30 01:24:26.425525 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425255    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-dshm\") pod \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") "
Oct 30 01:24:26.425525 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425293    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-kube-api-access-rd99z\" (UniqueName: \"kubernetes.io/projected/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-kube-api-access-rd99z\") pod \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") "
Oct 30 01:24:26.425525 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425328    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-tmp-dir\") pod \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") "
Oct 30 01:24:26.425525 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425386    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-tmp-dir\") pod \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") "
Oct 30 01:24:26.425525 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425414    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-dshm\") pod \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") "
Oct 30 01:24:26.425525 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425440    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-model-cache\") pod \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") "
Oct 30 01:24:26.425525 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425472    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-tls-certs\") pod \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") "
Oct 30 01:24:26.425935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425526    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-model-cache\") pod \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") "
Oct 30 01:24:26.425935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425601    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf04d49-5a22-4e28-b566-a3cb4306ad08-kube-api-access-k5ffl\" (UniqueName: \"kubernetes.io/projected/dcf04d49-5a22-4e28-b566-a3cb4306ad08-kube-api-access-k5ffl\") pod \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") "
Oct 30 01:24:26.425935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425635    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-kube-api-access-b46rt\" (UniqueName: \"kubernetes.io/projected/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-kube-api-access-b46rt\") pod \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") "
Oct 30 01:24:26.425935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425680    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-model-cache\") pod \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") "
Oct 30 01:24:26.425935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425742    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-home\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-home\") pod \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") "
Oct 30 01:24:26.425935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425772    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-dshm\") pod \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") "
Oct 30 01:24:26.425935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425891    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-tmp-dir\") pod \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\" (UID: \"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b\") "
Oct 30 01:24:26.425935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425920    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-tls-certs\") pod \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\" (UID: \"3c7a9b83-c1a2-4460-8bad-5f3967a8d159\") "
Oct 30 01:24:26.426336 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425984    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcf04d49-5a22-4e28-b566-a3cb4306ad08-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf04d49-5a22-4e28-b566-a3cb4306ad08-tls-certs\") pod \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\" (UID: \"dcf04d49-5a22-4e28-b566-a3cb4306ad08\") "
Oct 30 01:24:26.427140 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.426582    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2967eaae-3210-4602-8b3f-f64ff80919cd-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.427140 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.426606    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2967eaae-3210-4602-8b3f-f64ff80919cd-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.427140 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.426622    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6flc\" (UniqueName: \"kubernetes.io/projected/2967eaae-3210-4602-8b3f-f64ff80919cd-kube-api-access-k6flc\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.427140 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425464    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-home" pod "3c7a9b83-c1a2-4460-8bad-5f3967a8d159" (UID: "3c7a9b83-c1a2-4460-8bad-5f3967a8d159"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:26.427140 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425647    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-tmp-dir" pod "3c7a9b83-c1a2-4460-8bad-5f3967a8d159" (UID: "3c7a9b83-c1a2-4460-8bad-5f3967a8d159"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:26.427140 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.425838    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-home" pod "dcf04d49-5a22-4e28-b566-a3cb4306ad08" (UID: "dcf04d49-5a22-4e28-b566-a3cb4306ad08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:26.427140 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.426876    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-tmp-dir" pod "2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" (UID: "2b5372b9-bd56-47d3-8e31-28bb7aa83e9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:26.427140 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.427111    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-tmp-dir" pod "dcf04d49-5a22-4e28-b566-a3cb4306ad08" (UID: "dcf04d49-5a22-4e28-b566-a3cb4306ad08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:26.427695 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.427492    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-model-cache" pod "2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" (UID: "2b5372b9-bd56-47d3-8e31-28bb7aa83e9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:26.427695 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.427538    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-model-cache" pod "dcf04d49-5a22-4e28-b566-a3cb4306ad08" (UID: "dcf04d49-5a22-4e28-b566-a3cb4306ad08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:26.428040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.427890    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-model-cache" pod "3c7a9b83-c1a2-4460-8bad-5f3967a8d159" (UID: "3c7a9b83-c1a2-4460-8bad-5f3967a8d159"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:26.428413 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.428378    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-home" pod "2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" (UID: "2b5372b9-bd56-47d3-8e31-28bb7aa83e9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:26.428676 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.428655    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-kube-api-access-rd99z" pod "3c7a9b83-c1a2-4460-8bad-5f3967a8d159" (UID: "3c7a9b83-c1a2-4460-8bad-5f3967a8d159"). InnerVolumeSpecName "kube-api-access-rd99z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:26.429188 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.429162    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf04d49-5a22-4e28-b566-a3cb4306ad08-kube-api-access-k5ffl" pod "dcf04d49-5a22-4e28-b566-a3cb4306ad08" (UID: "dcf04d49-5a22-4e28-b566-a3cb4306ad08"). InnerVolumeSpecName "kube-api-access-k5ffl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:26.429283 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.429239    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-dshm" pod "2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" (UID: "2b5372b9-bd56-47d3-8e31-28bb7aa83e9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:26.433078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.433043    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-dshm" pod "3c7a9b83-c1a2-4460-8bad-5f3967a8d159" (UID: "3c7a9b83-c1a2-4460-8bad-5f3967a8d159"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:26.433078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.433049    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-tls-certs" pod "2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" (UID: "2b5372b9-bd56-47d3-8e31-28bb7aa83e9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:26.433078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.433060    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-tls-certs" pod "3c7a9b83-c1a2-4460-8bad-5f3967a8d159" (UID: "3c7a9b83-c1a2-4460-8bad-5f3967a8d159"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:26.433078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.433070    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-dshm" pod "dcf04d49-5a22-4e28-b566-a3cb4306ad08" (UID: "dcf04d49-5a22-4e28-b566-a3cb4306ad08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:26.434339 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.434315    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-kube-api-access-b46rt" pod "2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" (UID: "2b5372b9-bd56-47d3-8e31-28bb7aa83e9b"). InnerVolumeSpecName "kube-api-access-b46rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:26.434634 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.434617    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf04d49-5a22-4e28-b566-a3cb4306ad08-tls-certs" pod "dcf04d49-5a22-4e28-b566-a3cb4306ad08" (UID: "dcf04d49-5a22-4e28-b566-a3cb4306ad08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:26.527779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.527668    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.527779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.527697    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.527779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.527736    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.527779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.527746    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd99z\" (UniqueName: \"kubernetes.io/projected/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-kube-api-access-rd99z\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.527779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.527755    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.527779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.527768    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.527779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.527778    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.527779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.527786    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.528328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.527796    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.528328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.527805    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.528328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.527813    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5ffl\" (UniqueName: \"kubernetes.io/projected/dcf04d49-5a22-4e28-b566-a3cb4306ad08-kube-api-access-k5ffl\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.528328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.527822    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b46rt\" (UniqueName: \"kubernetes.io/projected/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-kube-api-access-b46rt\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.528328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.527830    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.528328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.527840    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.528328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.527853    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf04d49-5a22-4e28-b566-a3cb4306ad08-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.528328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.527864    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.528328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.527875    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c7a9b83-c1a2-4460-8bad-5f3967a8d159-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.528328 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.527882    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf04d49-5a22-4e28-b566-a3cb4306ad08-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.744073 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.744017    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:24:26.909720 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.909672    6099 generic.go:356] "Generic (PLEG): container finished" podID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerID="3a980ce7dee14d9664cf0b8ecf5377a3ea3f8c7b7204da49d9126599e400b2fe" exitCode=0
Oct 30 01:24:26.910196 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.909731    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"
Oct 30 01:24:26.910196 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.909744    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" event={"ID":"dcf04d49-5a22-4e28-b566-a3cb4306ad08","Type":"ContainerDied","Data":"3a980ce7dee14d9664cf0b8ecf5377a3ea3f8c7b7204da49d9126599e400b2fe"}
Oct 30 01:24:26.910196 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.909797    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g" event={"ID":"dcf04d49-5a22-4e28-b566-a3cb4306ad08","Type":"ContainerDied","Data":"d9c6358d24164764053764e331391b7c1cf26727d7909fba96245fd4abcdd966"}
Oct 30 01:24:26.910196 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.909822    6099 scope.go:122] "RemoveContainer" containerID="3a980ce7dee14d9664cf0b8ecf5377a3ea3f8c7b7204da49d9126599e400b2fe"
Oct 30 01:24:26.911392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.911360    6099 generic.go:356] "Generic (PLEG): container finished" podID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerID="d30d5c9bd2d3de754c957ed0667336034a8bfa55620acd2e9f34ccebae69b3b0" exitCode=0
Oct 30 01:24:26.911528 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.911396    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"
Oct 30 01:24:26.911528 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.911501    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" event={"ID":"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b","Type":"ContainerDied","Data":"d30d5c9bd2d3de754c957ed0667336034a8bfa55620acd2e9f34ccebae69b3b0"}
Oct 30 01:24:26.911660 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.911533    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29" event={"ID":"2b5372b9-bd56-47d3-8e31-28bb7aa83e9b","Type":"ContainerDied","Data":"4561ef4c6e544ac676c1004ba91caeea411658c887ecf9e46448bf98414f0f77"}
Oct 30 01:24:26.915031 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.915008    6099 generic.go:356] "Generic (PLEG): container finished" podID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerID="69980549eeb11bd271e6b6b7cbcf94319e08287456d151e919132bb6a3e6cd65" exitCode=0
Oct 30 01:24:26.915150 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.915041    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"
Oct 30 01:24:26.915150 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.915086    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" event={"ID":"3c7a9b83-c1a2-4460-8bad-5f3967a8d159","Type":"ContainerDied","Data":"69980549eeb11bd271e6b6b7cbcf94319e08287456d151e919132bb6a3e6cd65"}
Oct 30 01:24:26.915150 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.915122    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh" event={"ID":"3c7a9b83-c1a2-4460-8bad-5f3967a8d159","Type":"ContainerDied","Data":"77bfb04dbce26b4eadb94e18a50cc920f8d4836310de84fa599233d98abe07d4"}
Oct 30 01:24:26.917228 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.917048    6099 generic.go:356] "Generic (PLEG): container finished" podID="6098fd9c-5bb6-471b-8566-f845870c93c8" containerID="ccb845bf03ebdf4918654d6745532b1d1aa0006c95dfa11a7ebdca4bbb17d39a" exitCode=0
Oct 30 01:24:26.917228 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.917126    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht" event={"ID":"6098fd9c-5bb6-471b-8566-f845870c93c8","Type":"ContainerDied","Data":"ccb845bf03ebdf4918654d6745532b1d1aa0006c95dfa11a7ebdca4bbb17d39a"}
Oct 30 01:24:26.931023 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.930994    6099 generic.go:356] "Generic (PLEG): container finished" podID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerID="0f6befec8563c3fe8f865e5889831cf8c4558ba8addff016128fecd0772d8e0c" exitCode=0
Oct 30 01:24:26.931141 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.931056    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" event={"ID":"2967eaae-3210-4602-8b3f-f64ff80919cd","Type":"ContainerDied","Data":"0f6befec8563c3fe8f865e5889831cf8c4558ba8addff016128fecd0772d8e0c"}
Oct 30 01:24:26.931141 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.931081    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v" event={"ID":"2967eaae-3210-4602-8b3f-f64ff80919cd","Type":"ContainerDied","Data":"ea3b9773e243bea1f0a97e8d341abd96a3c7215979aab9fded4ac68e5349689c"}
Oct 30 01:24:26.931272 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.931170    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"
Oct 30 01:24:26.949610 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.949588    6099 scope.go:122] "RemoveContainer" containerID="3a980ce7dee14d9664cf0b8ecf5377a3ea3f8c7b7204da49d9126599e400b2fe"
Oct 30 01:24:26.949946 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:24:26.949926    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a980ce7dee14d9664cf0b8ecf5377a3ea3f8c7b7204da49d9126599e400b2fe\": container with ID starting with 3a980ce7dee14d9664cf0b8ecf5377a3ea3f8c7b7204da49d9126599e400b2fe not found: ID does not exist" containerID="3a980ce7dee14d9664cf0b8ecf5377a3ea3f8c7b7204da49d9126599e400b2fe"
Oct 30 01:24:26.950035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.949959    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a980ce7dee14d9664cf0b8ecf5377a3ea3f8c7b7204da49d9126599e400b2fe"} err="failed to get container status \"3a980ce7dee14d9664cf0b8ecf5377a3ea3f8c7b7204da49d9126599e400b2fe\": rpc error: code = NotFound desc = could not find container \"3a980ce7dee14d9664cf0b8ecf5377a3ea3f8c7b7204da49d9126599e400b2fe\": container with ID starting with 3a980ce7dee14d9664cf0b8ecf5377a3ea3f8c7b7204da49d9126599e400b2fe not found: ID does not exist"
Oct 30 01:24:26.950035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.949982    6099 scope.go:122] "RemoveContainer" containerID="d30d5c9bd2d3de754c957ed0667336034a8bfa55620acd2e9f34ccebae69b3b0"
Oct 30 01:24:26.959521 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.959493    6099 scope.go:122] "RemoveContainer" containerID="d30d5c9bd2d3de754c957ed0667336034a8bfa55620acd2e9f34ccebae69b3b0"
Oct 30 01:24:26.959823 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:24:26.959795    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30d5c9bd2d3de754c957ed0667336034a8bfa55620acd2e9f34ccebae69b3b0\": container with ID starting with d30d5c9bd2d3de754c957ed0667336034a8bfa55620acd2e9f34ccebae69b3b0 not found: ID does not exist" containerID="d30d5c9bd2d3de754c957ed0667336034a8bfa55620acd2e9f34ccebae69b3b0"
Oct 30 01:24:26.959882 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.959824    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30d5c9bd2d3de754c957ed0667336034a8bfa55620acd2e9f34ccebae69b3b0"} err="failed to get container status \"d30d5c9bd2d3de754c957ed0667336034a8bfa55620acd2e9f34ccebae69b3b0\": rpc error: code = NotFound desc = could not find container \"d30d5c9bd2d3de754c957ed0667336034a8bfa55620acd2e9f34ccebae69b3b0\": container with ID starting with d30d5c9bd2d3de754c957ed0667336034a8bfa55620acd2e9f34ccebae69b3b0 not found: ID does not exist"
Oct 30 01:24:26.959882 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.959840    6099 scope.go:122] "RemoveContainer" containerID="69980549eeb11bd271e6b6b7cbcf94319e08287456d151e919132bb6a3e6cd65"
Oct 30 01:24:26.970225 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.970204    6099 scope.go:122] "RemoveContainer" containerID="69980549eeb11bd271e6b6b7cbcf94319e08287456d151e919132bb6a3e6cd65"
Oct 30 01:24:26.970528 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:24:26.970506    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69980549eeb11bd271e6b6b7cbcf94319e08287456d151e919132bb6a3e6cd65\": container with ID starting with 69980549eeb11bd271e6b6b7cbcf94319e08287456d151e919132bb6a3e6cd65 not found: ID does not exist" containerID="69980549eeb11bd271e6b6b7cbcf94319e08287456d151e919132bb6a3e6cd65"
Oct 30 01:24:26.970671 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.970539    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69980549eeb11bd271e6b6b7cbcf94319e08287456d151e919132bb6a3e6cd65"} err="failed to get container status \"69980549eeb11bd271e6b6b7cbcf94319e08287456d151e919132bb6a3e6cd65\": rpc error: code = NotFound desc = could not find container \"69980549eeb11bd271e6b6b7cbcf94319e08287456d151e919132bb6a3e6cd65\": container with ID starting with 69980549eeb11bd271e6b6b7cbcf94319e08287456d151e919132bb6a3e6cd65 not found: ID does not exist"
Oct 30 01:24:26.970671 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.970581    6099 scope.go:122] "RemoveContainer" containerID="0f6befec8563c3fe8f865e5889831cf8c4558ba8addff016128fecd0772d8e0c"
Oct 30 01:24:26.979949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.979929    6099 scope.go:122] "RemoveContainer" containerID="0f6befec8563c3fe8f865e5889831cf8c4558ba8addff016128fecd0772d8e0c"
Oct 30 01:24:26.980239 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:24:26.980221    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f6befec8563c3fe8f865e5889831cf8c4558ba8addff016128fecd0772d8e0c\": container with ID starting with 0f6befec8563c3fe8f865e5889831cf8c4558ba8addff016128fecd0772d8e0c not found: ID does not exist" containerID="0f6befec8563c3fe8f865e5889831cf8c4558ba8addff016128fecd0772d8e0c"
Oct 30 01:24:26.980304 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:26.980246    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f6befec8563c3fe8f865e5889831cf8c4558ba8addff016128fecd0772d8e0c"} err="failed to get container status \"0f6befec8563c3fe8f865e5889831cf8c4558ba8addff016128fecd0772d8e0c\": rpc error: code = NotFound desc = could not find container \"0f6befec8563c3fe8f865e5889831cf8c4558ba8addff016128fecd0772d8e0c\": container with ID starting with 0f6befec8563c3fe8f865e5889831cf8c4558ba8addff016128fecd0772d8e0c not found: ID does not exist"
Oct 30 01:24:27.044487 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:27.044456    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"]
Oct 30 01:24:27.051578 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:27.051544    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/distinct-w6-5062f6-kserve-5d7fdc86bb-b8g2g"]
Oct 30 01:24:27.091956 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:27.091923    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"]
Oct 30 01:24:27.105582 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:27.105552    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/premium-w6-5062f6-kserve-bf6d5d887-2bvlh"]
Oct 30 01:24:27.143479 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:27.143445    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"]
Oct 30 01:24:27.155904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:27.155873    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/distinct-2-w6-5062f6-kserve-d9f7ff789-tmx8v"]
Oct 30 01:24:27.198055 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:27.197976    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"]
Oct 30 01:24:27.210507 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:27.210473    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5062f6/simulator-w6-5062f6-kserve-55764c85bb-xsv29"]
Oct 30 01:24:28.087133 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:28.087108    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht"
Oct 30 01:24:28.159730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:28.159680    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6098fd9c-5bb6-471b-8566-f845870c93c8-kube-api-access-wj7nn\" (UniqueName: \"kubernetes.io/projected/6098fd9c-5bb6-471b-8566-f845870c93c8-kube-api-access-wj7nn\") pod \"6098fd9c-5bb6-471b-8566-f845870c93c8\" (UID: \"6098fd9c-5bb6-471b-8566-f845870c93c8\") "
Oct 30 01:24:28.159909 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:28.159884    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6098fd9c-5bb6-471b-8566-f845870c93c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6098fd9c-5bb6-471b-8566-f845870c93c8-maas-api-xks-ca\") pod \"6098fd9c-5bb6-471b-8566-f845870c93c8\" (UID: \"6098fd9c-5bb6-471b-8566-f845870c93c8\") "
Oct 30 01:24:28.159984 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:28.159967    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6098fd9c-5bb6-471b-8566-f845870c93c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6098fd9c-5bb6-471b-8566-f845870c93c8-maas-api-openshift-ca\") pod \"6098fd9c-5bb6-471b-8566-f845870c93c8\" (UID: \"6098fd9c-5bb6-471b-8566-f845870c93c8\") "
Oct 30 01:24:28.161662 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:28.160382    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6098fd9c-5bb6-471b-8566-f845870c93c8-maas-api-openshift-ca" pod "6098fd9c-5bb6-471b-8566-f845870c93c8" (UID: "6098fd9c-5bb6-471b-8566-f845870c93c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:28.162028 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:28.162005    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6098fd9c-5bb6-471b-8566-f845870c93c8-maas-api-xks-ca" pod "6098fd9c-5bb6-471b-8566-f845870c93c8" (UID: "6098fd9c-5bb6-471b-8566-f845870c93c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:28.163056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:28.163030    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6098fd9c-5bb6-471b-8566-f845870c93c8-kube-api-access-wj7nn" pod "6098fd9c-5bb6-471b-8566-f845870c93c8" (UID: "6098fd9c-5bb6-471b-8566-f845870c93c8"). InnerVolumeSpecName "kube-api-access-wj7nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:28.261251 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:28.261165    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" path="/var/lib/kubelet/pods/2967eaae-3210-4602-8b3f-f64ff80919cd/volumes"
Oct 30 01:24:28.261420 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:28.261280    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6098fd9c-5bb6-471b-8566-f845870c93c8-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:28.261420 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:28.261303    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6098fd9c-5bb6-471b-8566-f845870c93c8-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:28.261420 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:28.261318    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj7nn\" (UniqueName: \"kubernetes.io/projected/6098fd9c-5bb6-471b-8566-f845870c93c8-kube-api-access-wj7nn\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:28.261574 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:28.261560    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" path="/var/lib/kubelet/pods/2b5372b9-bd56-47d3-8e31-28bb7aa83e9b/volumes"
Oct 30 01:24:28.261984 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:28.261967    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" path="/var/lib/kubelet/pods/3c7a9b83-c1a2-4460-8bad-5f3967a8d159/volumes"
Oct 30 01:24:28.262662 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:28.262645    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" path="/var/lib/kubelet/pods/dcf04d49-5a22-4e28-b566-a3cb4306ad08/volumes"
Oct 30 01:24:28.948781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:28.948679    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht"
Oct 30 01:24:28.948781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:28.948731    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht" event={"ID":"6098fd9c-5bb6-471b-8566-f845870c93c8","Type":"ContainerDied","Data":"d7313d2bcb12447aaa8c1edf0b48be5989f97ccdebc1513f8445bd6cbdeacf06"}
Oct 30 01:24:28.948781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:28.948782    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7313d2bcb12447aaa8c1edf0b48be5989f97ccdebc1513f8445bd6cbdeacf06"
Oct 30 01:24:30.240346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.240313    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x"]
Oct 30 01:24:30.240845 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.240830    6099 cpu_manager.go:414] "containerMap: removing container" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main"
Oct 30 01:24:30.240845 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.240845    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main"
Oct 30 01:24:30.240951 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.240892    6099 cpu_manager.go:414] "containerMap: removing container" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main"
Oct 30 01:24:30.240951 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.240897    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main"
Oct 30 01:24:30.240951 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.240906    6099 cpu_manager.go:414] "containerMap: removing container" podUID="6098fd9c-5bb6-471b-8566-f845870c93c8" containerName="revoke-keys"
Oct 30 01:24:30.240951 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.240911    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6098fd9c-5bb6-471b-8566-f845870c93c8" containerName="revoke-keys"
Oct 30 01:24:30.240951 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.240927    6099 cpu_manager.go:414] "containerMap: removing container" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main"
Oct 30 01:24:30.240951 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.240932    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main"
Oct 30 01:24:30.240951 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.240943    6099 cpu_manager.go:414] "containerMap: removing container" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main"
Oct 30 01:24:30.240951 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.240947    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main"
Oct 30 01:24:30.241210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.241031    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c7a9b83-c1a2-4460-8bad-5f3967a8d159" containerName="main"
Oct 30 01:24:30.241210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.241046    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf04d49-5a22-4e28-b566-a3cb4306ad08" containerName="main"
Oct 30 01:24:30.241210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.241052    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6098fd9c-5bb6-471b-8566-f845870c93c8" containerName="revoke-keys"
Oct 30 01:24:30.241210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.241061    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2967eaae-3210-4602-8b3f-f64ff80919cd" containerName="main"
Oct 30 01:24:30.241210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.241069    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5372b9-bd56-47d3-8e31-28bb7aa83e9b" containerName="main"
Oct 30 01:24:30.245405 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.245388    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x"
Oct 30 01:24:30.266949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.266922    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x"]
Oct 30 01:24:30.275955 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.275930    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:24:30.276081 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.276036    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:24:30.387460 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.387419    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13935af2-npw5x\" (UID: \"f597c60b-9751-46a4-b4da-7b82c6fa4b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x"
Oct 30 01:24:30.387614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.387493    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13935af2-npw5x\" (UID: \"f597c60b-9751-46a4-b4da-7b82c6fa4b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x"
Oct 30 01:24:30.387677 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.387604    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2vq2\" (UniqueName: \"kubernetes.io/projected/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-kube-api-access-k2vq2\") pod \"maas-api-revoke-sub-13935af2-npw5x\" (UID: \"f597c60b-9751-46a4-b4da-7b82c6fa4b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x"
Oct 30 01:24:30.489110 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.489074    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13935af2-npw5x\" (UID: \"f597c60b-9751-46a4-b4da-7b82c6fa4b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x"
Oct 30 01:24:30.489268 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.489152    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13935af2-npw5x\" (UID: \"f597c60b-9751-46a4-b4da-7b82c6fa4b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x"
Oct 30 01:24:30.489268 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.489208    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2vq2\" (UniqueName: \"kubernetes.io/projected/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-kube-api-access-k2vq2\") pod \"maas-api-revoke-sub-13935af2-npw5x\" (UID: \"f597c60b-9751-46a4-b4da-7b82c6fa4b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x"
Oct 30 01:24:30.489850 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.489825    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13935af2-npw5x\" (UID: \"f597c60b-9751-46a4-b4da-7b82c6fa4b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x"
Oct 30 01:24:30.491486 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.491442    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13935af2-npw5x\" (UID: \"f597c60b-9751-46a4-b4da-7b82c6fa4b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x"
Oct 30 01:24:30.513214 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.513191    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2vq2\" (UniqueName: \"kubernetes.io/projected/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-kube-api-access-k2vq2\") pod \"maas-api-revoke-sub-13935af2-npw5x\" (UID: \"f597c60b-9751-46a4-b4da-7b82c6fa4b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x"
Oct 30 01:24:30.559624 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.559596    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x"
Oct 30 01:24:30.863896 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.863864    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x"]
Oct 30 01:24:30.864213 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:24:30.864184    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf597c60b_9751_46a4_b4da_7b82c6fa4b1e.slice/crio-7cec41016e48c9cf1463a4c3e214a95cc85c3ae2cf7409e67abd8b323962f679 WatchSource:0}: Error finding container 7cec41016e48c9cf1463a4c3e214a95cc85c3ae2cf7409e67abd8b323962f679: Status 404 returned error can't find the container with id 7cec41016e48c9cf1463a4c3e214a95cc85c3ae2cf7409e67abd8b323962f679
Oct 30 01:24:30.963565 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:30.963530    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x" event={"ID":"f597c60b-9751-46a4-b4da-7b82c6fa4b1e","Type":"ContainerStarted","Data":"7cec41016e48c9cf1463a4c3e214a95cc85c3ae2cf7409e67abd8b323962f679"}
Oct 30 01:24:31.973418 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:31.973335    6099 generic.go:356] "Generic (PLEG): container finished" podID="f597c60b-9751-46a4-b4da-7b82c6fa4b1e" containerID="4ca5e5a144ee25f920fa4ebb88efda71ba202b289091619d207897614bfff917" exitCode=0
Oct 30 01:24:31.973835 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:31.973415    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x" event={"ID":"f597c60b-9751-46a4-b4da-7b82c6fa4b1e","Type":"ContainerDied","Data":"4ca5e5a144ee25f920fa4ebb88efda71ba202b289091619d207897614bfff917"}
Oct 30 01:24:33.117382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:33.117351    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x"
Oct 30 01:24:33.230238 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:33.230201    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-kube-api-access-k2vq2\" (UniqueName: \"kubernetes.io/projected/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-kube-api-access-k2vq2\") pod \"f597c60b-9751-46a4-b4da-7b82c6fa4b1e\" (UID: \"f597c60b-9751-46a4-b4da-7b82c6fa4b1e\") "
Oct 30 01:24:33.230456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:33.230252    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-maas-api-xks-ca\") pod \"f597c60b-9751-46a4-b4da-7b82c6fa4b1e\" (UID: \"f597c60b-9751-46a4-b4da-7b82c6fa4b1e\") "
Oct 30 01:24:33.230456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:33.230365    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-maas-api-openshift-ca\") pod \"f597c60b-9751-46a4-b4da-7b82c6fa4b1e\" (UID: \"f597c60b-9751-46a4-b4da-7b82c6fa4b1e\") "
Oct 30 01:24:33.230774 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:33.230741    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-maas-api-openshift-ca" pod "f597c60b-9751-46a4-b4da-7b82c6fa4b1e" (UID: "f597c60b-9751-46a4-b4da-7b82c6fa4b1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:33.232405 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:33.232382    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-maas-api-xks-ca" pod "f597c60b-9751-46a4-b4da-7b82c6fa4b1e" (UID: "f597c60b-9751-46a4-b4da-7b82c6fa4b1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:33.232512 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:33.232386    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-kube-api-access-k2vq2" pod "f597c60b-9751-46a4-b4da-7b82c6fa4b1e" (UID: "f597c60b-9751-46a4-b4da-7b82c6fa4b1e"). InnerVolumeSpecName "kube-api-access-k2vq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:33.331965 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:33.331928    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2vq2\" (UniqueName: \"kubernetes.io/projected/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-kube-api-access-k2vq2\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:33.331965 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:33.331959    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:33.331965 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:33.331968    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f597c60b-9751-46a4-b4da-7b82c6fa4b1e-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:33.989144 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:33.989114    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x"
Oct 30 01:24:33.989321 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:33.989144    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x" event={"ID":"f597c60b-9751-46a4-b4da-7b82c6fa4b1e","Type":"ContainerDied","Data":"7cec41016e48c9cf1463a4c3e214a95cc85c3ae2cf7409e67abd8b323962f679"}
Oct 30 01:24:33.989321 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:33.989197    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cec41016e48c9cf1463a4c3e214a95cc85c3ae2cf7409e67abd8b323962f679"
Oct 30 01:24:39.418736 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:39.418690    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c"]
Oct 30 01:24:39.419223 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:39.418887    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c" podUID="48894977-815d-4b0d-be69-c9d7f29aa1a4" containerName="payload-processing" containerID="cri-o://c3db3e0512be4f1670bbe0e98ad3efa0bb701550fc9f792a6ecb2b02d56dd3db" gracePeriod=30
Oct 30 01:24:39.697731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:39.697674    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c"
Oct 30 01:24:39.806518 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:39.805434    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48894977-815d-4b0d-be69-c9d7f29aa1a4-kube-api-access-rwp2m\" (UniqueName: \"kubernetes.io/projected/48894977-815d-4b0d-be69-c9d7f29aa1a4-kube-api-access-rwp2m\") pod \"48894977-815d-4b0d-be69-c9d7f29aa1a4\" (UID: \"48894977-815d-4b0d-be69-c9d7f29aa1a4\") "
Oct 30 01:24:39.806518 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:39.806189    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48894977-815d-4b0d-be69-c9d7f29aa1a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48894977-815d-4b0d-be69-c9d7f29aa1a4-plugins-config-volume\") pod \"48894977-815d-4b0d-be69-c9d7f29aa1a4\" (UID: \"48894977-815d-4b0d-be69-c9d7f29aa1a4\") "
Oct 30 01:24:39.806781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:39.806756    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48894977-815d-4b0d-be69-c9d7f29aa1a4-plugins-config-volume" pod "48894977-815d-4b0d-be69-c9d7f29aa1a4" (UID: "48894977-815d-4b0d-be69-c9d7f29aa1a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:39.807327 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:39.807302    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48894977-815d-4b0d-be69-c9d7f29aa1a4-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:39.809650 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:39.809620    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48894977-815d-4b0d-be69-c9d7f29aa1a4-kube-api-access-rwp2m" pod "48894977-815d-4b0d-be69-c9d7f29aa1a4" (UID: "48894977-815d-4b0d-be69-c9d7f29aa1a4"). InnerVolumeSpecName "kube-api-access-rwp2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:39.908318 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:39.908284    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwp2m\" (UniqueName: \"kubernetes.io/projected/48894977-815d-4b0d-be69-c9d7f29aa1a4-kube-api-access-rwp2m\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:39.981697 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:39.981601    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht"]
Oct 30 01:24:39.983754 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:39.983728    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3e9921-2kbht"]
Oct 30 01:24:40.031139 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:40.031105    6099 generic.go:356] "Generic (PLEG): container finished" podID="48894977-815d-4b0d-be69-c9d7f29aa1a4" containerID="c3db3e0512be4f1670bbe0e98ad3efa0bb701550fc9f792a6ecb2b02d56dd3db" exitCode=0
Oct 30 01:24:40.031326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:40.031151    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c"
Oct 30 01:24:40.031326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:40.031188    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c" event={"ID":"48894977-815d-4b0d-be69-c9d7f29aa1a4","Type":"ContainerDied","Data":"c3db3e0512be4f1670bbe0e98ad3efa0bb701550fc9f792a6ecb2b02d56dd3db"}
Oct 30 01:24:40.031326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:40.031238    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c" event={"ID":"48894977-815d-4b0d-be69-c9d7f29aa1a4","Type":"ContainerDied","Data":"e3d2f8e6b5a065a4770f8239ad68920470c094c67f22ff83ab40948c3a7f17ba"}
Oct 30 01:24:40.031326 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:40.031261    6099 scope.go:122] "RemoveContainer" containerID="c3db3e0512be4f1670bbe0e98ad3efa0bb701550fc9f792a6ecb2b02d56dd3db"
Oct 30 01:24:40.041983 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:40.041960    6099 scope.go:122] "RemoveContainer" containerID="c3db3e0512be4f1670bbe0e98ad3efa0bb701550fc9f792a6ecb2b02d56dd3db"
Oct 30 01:24:40.042250 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:24:40.042233    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3db3e0512be4f1670bbe0e98ad3efa0bb701550fc9f792a6ecb2b02d56dd3db\": container with ID starting with c3db3e0512be4f1670bbe0e98ad3efa0bb701550fc9f792a6ecb2b02d56dd3db not found: ID does not exist" containerID="c3db3e0512be4f1670bbe0e98ad3efa0bb701550fc9f792a6ecb2b02d56dd3db"
Oct 30 01:24:40.042306 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:40.042257    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3db3e0512be4f1670bbe0e98ad3efa0bb701550fc9f792a6ecb2b02d56dd3db"} err="failed to get container status \"c3db3e0512be4f1670bbe0e98ad3efa0bb701550fc9f792a6ecb2b02d56dd3db\": rpc error: code = NotFound desc = could not find container \"c3db3e0512be4f1670bbe0e98ad3efa0bb701550fc9f792a6ecb2b02d56dd3db\": container with ID starting with c3db3e0512be4f1670bbe0e98ad3efa0bb701550fc9f792a6ecb2b02d56dd3db not found: ID does not exist"
Oct 30 01:24:40.082432 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:40.082402    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c"]
Oct 30 01:24:40.085476 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:40.085452    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e9921-6555c58b98-k6f8c"]
Oct 30 01:24:40.261192 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:40.261117    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48894977-815d-4b0d-be69-c9d7f29aa1a4" path="/var/lib/kubelet/pods/48894977-815d-4b0d-be69-c9d7f29aa1a4/volumes"
Oct 30 01:24:40.261566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:40.261552    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6098fd9c-5bb6-471b-8566-f845870c93c8" path="/var/lib/kubelet/pods/6098fd9c-5bb6-471b-8566-f845870c93c8/volumes"
Oct 30 01:24:41.065654 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.065621    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6"]
Oct 30 01:24:41.066226 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.066206    6099 cpu_manager.go:414] "containerMap: removing container" podUID="f597c60b-9751-46a4-b4da-7b82c6fa4b1e" containerName="revoke-keys"
Oct 30 01:24:41.066273 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.066230    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f597c60b-9751-46a4-b4da-7b82c6fa4b1e" containerName="revoke-keys"
Oct 30 01:24:41.066311 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.066305    6099 cpu_manager.go:414] "containerMap: removing container" podUID="48894977-815d-4b0d-be69-c9d7f29aa1a4" containerName="payload-processing"
Oct 30 01:24:41.066346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.066315    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48894977-815d-4b0d-be69-c9d7f29aa1a4" containerName="payload-processing"
Oct 30 01:24:41.066439 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.066429    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="48894977-815d-4b0d-be69-c9d7f29aa1a4" containerName="payload-processing"
Oct 30 01:24:41.066480 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.066444    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f597c60b-9751-46a4-b4da-7b82c6fa4b1e" containerName="revoke-keys"
Oct 30 01:24:41.071458 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.071441    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6"
Oct 30 01:24:41.089092 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.089063    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6"]
Oct 30 01:24:41.098080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.098060    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:24:41.098203 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.098101    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:24:41.121687 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.121662    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6\" (UID: \"ecd581e7-379c-4e62-aa82-15e76ad1ffc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6"
Oct 30 01:24:41.121844 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.121810    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6\" (UID: \"ecd581e7-379c-4e62-aa82-15e76ad1ffc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6"
Oct 30 01:24:41.121931 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.121862    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj77l\" (UniqueName: \"kubernetes.io/projected/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-kube-api-access-wj77l\") pod \"maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6\" (UID: \"ecd581e7-379c-4e62-aa82-15e76ad1ffc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6"
Oct 30 01:24:41.222621 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.222591    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6\" (UID: \"ecd581e7-379c-4e62-aa82-15e76ad1ffc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6"
Oct 30 01:24:41.222798 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.222688    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6\" (UID: \"ecd581e7-379c-4e62-aa82-15e76ad1ffc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6"
Oct 30 01:24:41.222798 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.222745    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj77l\" (UniqueName: \"kubernetes.io/projected/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-kube-api-access-wj77l\") pod \"maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6\" (UID: \"ecd581e7-379c-4e62-aa82-15e76ad1ffc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6"
Oct 30 01:24:41.223356 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.223334    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6\" (UID: \"ecd581e7-379c-4e62-aa82-15e76ad1ffc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6"
Oct 30 01:24:41.225057 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.225033    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6\" (UID: \"ecd581e7-379c-4e62-aa82-15e76ad1ffc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6"
Oct 30 01:24:41.245302 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.245278    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj77l\" (UniqueName: \"kubernetes.io/projected/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-kube-api-access-wj77l\") pod \"maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6\" (UID: \"ecd581e7-379c-4e62-aa82-15e76ad1ffc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6"
Oct 30 01:24:41.384929 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.384854    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6"
Oct 30 01:24:41.684318 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:41.684289    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6"]
Oct 30 01:24:41.684936 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:24:41.684910    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd581e7_379c_4e62_aa82_15e76ad1ffc2.slice/crio-4470aeceade9ebe91cbed6462ccc27bcbd8c0bef885efd081ca9e33bcde42753 WatchSource:0}: Error finding container 4470aeceade9ebe91cbed6462ccc27bcbd8c0bef885efd081ca9e33bcde42753: Status 404 returned error can't find the container with id 4470aeceade9ebe91cbed6462ccc27bcbd8c0bef885efd081ca9e33bcde42753
Oct 30 01:24:42.045775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:42.045743    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6" event={"ID":"ecd581e7-379c-4e62-aa82-15e76ad1ffc2","Type":"ContainerStarted","Data":"4470aeceade9ebe91cbed6462ccc27bcbd8c0bef885efd081ca9e33bcde42753"}
Oct 30 01:24:43.055122 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:43.055088    6099 generic.go:356] "Generic (PLEG): container finished" podID="ecd581e7-379c-4e62-aa82-15e76ad1ffc2" containerID="aad4d78276f671761a0763593022fce4296673fbbe9839f1e686688bcd10a478" exitCode=0
Oct 30 01:24:43.055492 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:43.055189    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6" event={"ID":"ecd581e7-379c-4e62-aa82-15e76ad1ffc2","Type":"ContainerDied","Data":"aad4d78276f671761a0763593022fce4296673fbbe9839f1e686688bcd10a478"}
Oct 30 01:24:44.206921 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:44.206893    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6"
Oct 30 01:24:44.257878 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:44.257845    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-kube-api-access-wj77l\" (UniqueName: \"kubernetes.io/projected/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-kube-api-access-wj77l\") pod \"ecd581e7-379c-4e62-aa82-15e76ad1ffc2\" (UID: \"ecd581e7-379c-4e62-aa82-15e76ad1ffc2\") "
Oct 30 01:24:44.258048 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:44.257894    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-maas-api-openshift-ca\") pod \"ecd581e7-379c-4e62-aa82-15e76ad1ffc2\" (UID: \"ecd581e7-379c-4e62-aa82-15e76ad1ffc2\") "
Oct 30 01:24:44.258048 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:44.257993    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-maas-api-xks-ca\") pod \"ecd581e7-379c-4e62-aa82-15e76ad1ffc2\" (UID: \"ecd581e7-379c-4e62-aa82-15e76ad1ffc2\") "
Oct 30 01:24:44.258304 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:44.258275    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-maas-api-openshift-ca" pod "ecd581e7-379c-4e62-aa82-15e76ad1ffc2" (UID: "ecd581e7-379c-4e62-aa82-15e76ad1ffc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:44.258566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:44.258543    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:44.261116 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:44.260502    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-kube-api-access-wj77l" pod "ecd581e7-379c-4e62-aa82-15e76ad1ffc2" (UID: "ecd581e7-379c-4e62-aa82-15e76ad1ffc2"). InnerVolumeSpecName "kube-api-access-wj77l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:44.261116 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:44.260756    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-maas-api-xks-ca" pod "ecd581e7-379c-4e62-aa82-15e76ad1ffc2" (UID: "ecd581e7-379c-4e62-aa82-15e76ad1ffc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:44.360183 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:44.360109    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj77l\" (UniqueName: \"kubernetes.io/projected/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-kube-api-access-wj77l\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:44.360183 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:44.360137    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd581e7-379c-4e62-aa82-15e76ad1ffc2-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:44.462758 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:44.462719    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:24:45.068917 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:45.068888    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6"
Oct 30 01:24:45.069080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:45.068929    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6" event={"ID":"ecd581e7-379c-4e62-aa82-15e76ad1ffc2","Type":"ContainerDied","Data":"4470aeceade9ebe91cbed6462ccc27bcbd8c0bef885efd081ca9e33bcde42753"}
Oct 30 01:24:45.069080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:45.068972    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4470aeceade9ebe91cbed6462ccc27bcbd8c0bef885efd081ca9e33bcde42753"
Oct 30 01:24:46.630352 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:24:46.630317    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:24:48.716371 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:48.716336    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2"]
Oct 30 01:24:48.716902 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:48.716887    6099 cpu_manager.go:414] "containerMap: removing container" podUID="ecd581e7-379c-4e62-aa82-15e76ad1ffc2" containerName="revoke-keys"
Oct 30 01:24:48.716948 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:48.716903    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd581e7-379c-4e62-aa82-15e76ad1ffc2" containerName="revoke-keys"
Oct 30 01:24:48.717026 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:48.717016    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd581e7-379c-4e62-aa82-15e76ad1ffc2" containerName="revoke-keys"
Oct 30 01:24:48.720662 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:48.720646    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2"
Oct 30 01:24:48.739866 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:48.739837    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2"]
Oct 30 01:24:48.799928 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:48.799898    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:24:48.800160 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:48.800090    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:24:48.839729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:48.836661    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2\" (UID: \"df0dbed8-9648-4bce-87e0-da0df2e2e8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2"
Oct 30 01:24:48.839729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:48.836839    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2\" (UID: \"df0dbed8-9648-4bce-87e0-da0df2e2e8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2"
Oct 30 01:24:48.839729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:48.836880    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kvps\" (UniqueName: \"kubernetes.io/projected/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-kube-api-access-9kvps\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2\" (UID: \"df0dbed8-9648-4bce-87e0-da0df2e2e8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2"
Oct 30 01:24:48.938391 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:48.938359    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2\" (UID: \"df0dbed8-9648-4bce-87e0-da0df2e2e8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2"
Oct 30 01:24:48.938576 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:48.938451    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2\" (UID: \"df0dbed8-9648-4bce-87e0-da0df2e2e8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2"
Oct 30 01:24:48.938576 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:48.938469    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kvps\" (UniqueName: \"kubernetes.io/projected/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-kube-api-access-9kvps\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2\" (UID: \"df0dbed8-9648-4bce-87e0-da0df2e2e8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2"
Oct 30 01:24:48.939204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:48.939176    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2\" (UID: \"df0dbed8-9648-4bce-87e0-da0df2e2e8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2"
Oct 30 01:24:48.940794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:48.940771    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2\" (UID: \"df0dbed8-9648-4bce-87e0-da0df2e2e8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2"
Oct 30 01:24:48.961243 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:48.961216    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kvps\" (UniqueName: \"kubernetes.io/projected/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-kube-api-access-9kvps\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2\" (UID: \"df0dbed8-9648-4bce-87e0-da0df2e2e8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2"
Oct 30 01:24:49.034281 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:49.034210    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2"
Oct 30 01:24:49.334486 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:49.334448    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2"]
Oct 30 01:24:49.334588 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:24:49.334539    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0dbed8_9648_4bce_87e0_da0df2e2e8d0.slice/crio-3dc42a8e48f1b657278434a3a25e6a7b8b68abb7fc31ea266325fc6122bd0633 WatchSource:0}: Error finding container 3dc42a8e48f1b657278434a3a25e6a7b8b68abb7fc31ea266325fc6122bd0633: Status 404 returned error can't find the container with id 3dc42a8e48f1b657278434a3a25e6a7b8b68abb7fc31ea266325fc6122bd0633
Oct 30 01:24:50.104784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:50.104751    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2" event={"ID":"df0dbed8-9648-4bce-87e0-da0df2e2e8d0","Type":"ContainerStarted","Data":"e097cb112793aa5b06fbd2f11b89c18d802a388902d923ffef184bbc86a1c9ae"}
Oct 30 01:24:50.105187 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:50.104807    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2" event={"ID":"df0dbed8-9648-4bce-87e0-da0df2e2e8d0","Type":"ContainerStarted","Data":"3dc42a8e48f1b657278434a3a25e6a7b8b68abb7fc31ea266325fc6122bd0633"}
Oct 30 01:24:50.146426 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:50.146380    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2" podStartSLOduration=2.146365886 podStartE2EDuration="2.146365886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:50.144308129 +0000 UTC m=+2984.001215885" watchObservedRunningTime="2026-10-30 01:24:50.146365886 +0000 UTC m=+2984.003273641"
Oct 30 01:24:51.103226 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:51.103190    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8"]
Oct 30 01:24:51.108814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:51.108785    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5062f6-7pjf8"]
Oct 30 01:24:51.113594 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:51.113566    6099 generic.go:356] "Generic (PLEG): container finished" podID="df0dbed8-9648-4bce-87e0-da0df2e2e8d0" containerID="e097cb112793aa5b06fbd2f11b89c18d802a388902d923ffef184bbc86a1c9ae" exitCode=0
Oct 30 01:24:51.113770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:51.113630    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2" event={"ID":"df0dbed8-9648-4bce-87e0-da0df2e2e8d0","Type":"ContainerDied","Data":"e097cb112793aa5b06fbd2f11b89c18d802a388902d923ffef184bbc86a1c9ae"}
Oct 30 01:24:52.262162 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:52.262140    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2"
Oct 30 01:24:52.262861 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:52.262839    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd3f9333-df3a-4838-801e-b45201f33f25" path="/var/lib/kubelet/pods/dd3f9333-df3a-4838-801e-b45201f33f25/volumes"
Oct 30 01:24:52.378221 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:52.378142    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-maas-api-openshift-ca\") pod \"df0dbed8-9648-4bce-87e0-da0df2e2e8d0\" (UID: \"df0dbed8-9648-4bce-87e0-da0df2e2e8d0\") "
Oct 30 01:24:52.378221 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:52.378203    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-kube-api-access-9kvps\" (UniqueName: \"kubernetes.io/projected/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-kube-api-access-9kvps\") pod \"df0dbed8-9648-4bce-87e0-da0df2e2e8d0\" (UID: \"df0dbed8-9648-4bce-87e0-da0df2e2e8d0\") "
Oct 30 01:24:52.378444 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:52.378339    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-maas-api-xks-ca\") pod \"df0dbed8-9648-4bce-87e0-da0df2e2e8d0\" (UID: \"df0dbed8-9648-4bce-87e0-da0df2e2e8d0\") "
Oct 30 01:24:52.378601 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:52.378562    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-maas-api-openshift-ca" pod "df0dbed8-9648-4bce-87e0-da0df2e2e8d0" (UID: "df0dbed8-9648-4bce-87e0-da0df2e2e8d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:52.379061 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:52.379041    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:52.380450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:52.380425    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-maas-api-xks-ca" pod "df0dbed8-9648-4bce-87e0-da0df2e2e8d0" (UID: "df0dbed8-9648-4bce-87e0-da0df2e2e8d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:52.380546 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:52.380486    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-kube-api-access-9kvps" pod "df0dbed8-9648-4bce-87e0-da0df2e2e8d0" (UID: "df0dbed8-9648-4bce-87e0-da0df2e2e8d0"). InnerVolumeSpecName "kube-api-access-9kvps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:52.479880 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:52.479848    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kvps\" (UniqueName: \"kubernetes.io/projected/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-kube-api-access-9kvps\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:52.479880 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:52.479878    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0dbed8-9648-4bce-87e0-da0df2e2e8d0-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:24:53.127481 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:53.127449    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2" event={"ID":"df0dbed8-9648-4bce-87e0-da0df2e2e8d0","Type":"ContainerDied","Data":"3dc42a8e48f1b657278434a3a25e6a7b8b68abb7fc31ea266325fc6122bd0633"}
Oct 30 01:24:53.127481 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:53.127484    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2"
Oct 30 01:24:53.127722 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:53.127518    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dc42a8e48f1b657278434a3a25e6a7b8b68abb7fc31ea266325fc6122bd0633"
Oct 30 01:24:56.656722 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:24:56.655790    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:25:04.164470 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:04.164435    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2"]
Oct 30 01:25:04.175950 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:04.175925    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-95492e-925h2"]
Oct 30 01:25:04.261884 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:04.261846    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df0dbed8-9648-4bce-87e0-da0df2e2e8d0" path="/var/lib/kubelet/pods/df0dbed8-9648-4bce-87e0-da0df2e2e8d0/volumes"
Oct 30 01:25:04.899476 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:04.899445    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6"]
Oct 30 01:25:04.905129 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:04.905099    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc3cb2-mfwp6"]
Oct 30 01:25:06.261266 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:06.261234    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecd581e7-379c-4e62-aa82-15e76ad1ffc2" path="/var/lib/kubelet/pods/ecd581e7-379c-4e62-aa82-15e76ad1ffc2/volumes"
Oct 30 01:25:06.926136 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:06.926102    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 01:25:06.938635 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:06.938608    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 01:25:09.490843 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:09.490807    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:25:11.319644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:11.319605    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"]
Oct 30 01:25:11.320289 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:11.319887    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q" podUID="5bbcb61d-3921-444a-bcc9-1b8382320f04" containerName="istio-proxy" containerID="cri-o://2167109bfbe4f5c784b335dc1d0b085c551bb3191aa2e85d38052c0654af4d3a" gracePeriod=30
Oct 30 01:25:12.980505 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:12.980471    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6"]
Oct 30 01:25:12.981141 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:12.981122    6099 cpu_manager.go:414] "containerMap: removing container" podUID="df0dbed8-9648-4bce-87e0-da0df2e2e8d0" containerName="revoke-keys"
Oct 30 01:25:12.981189 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:12.981141    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df0dbed8-9648-4bce-87e0-da0df2e2e8d0" containerName="revoke-keys"
Oct 30 01:25:12.981252 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:12.981240    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="df0dbed8-9648-4bce-87e0-da0df2e2e8d0" containerName="revoke-keys"
Oct 30 01:25:12.984716 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:12.984693    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6"
Oct 30 01:25:13.013386 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:13.013339    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:25:13.013386 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:13.013368    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:25:13.017264 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:13.017238    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6"]
Oct 30 01:25:13.061311 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:13.061285    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8p7k\" (UniqueName: \"kubernetes.io/projected/c52a67b5-78c3-4964-9f99-056cd1fd7a55-kube-api-access-f8p7k\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6\" (UID: \"c52a67b5-78c3-4964-9f99-056cd1fd7a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6"
Oct 30 01:25:13.061445 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:13.061363    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c52a67b5-78c3-4964-9f99-056cd1fd7a55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6\" (UID: \"c52a67b5-78c3-4964-9f99-056cd1fd7a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6"
Oct 30 01:25:13.061494 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:13.061477    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c52a67b5-78c3-4964-9f99-056cd1fd7a55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6\" (UID: \"c52a67b5-78c3-4964-9f99-056cd1fd7a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6"
Oct 30 01:25:13.162098 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:13.162068    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8p7k\" (UniqueName: \"kubernetes.io/projected/c52a67b5-78c3-4964-9f99-056cd1fd7a55-kube-api-access-f8p7k\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6\" (UID: \"c52a67b5-78c3-4964-9f99-056cd1fd7a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6"
Oct 30 01:25:13.162245 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:13.162133    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c52a67b5-78c3-4964-9f99-056cd1fd7a55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6\" (UID: \"c52a67b5-78c3-4964-9f99-056cd1fd7a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6"
Oct 30 01:25:13.162245 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:13.162189    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c52a67b5-78c3-4964-9f99-056cd1fd7a55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6\" (UID: \"c52a67b5-78c3-4964-9f99-056cd1fd7a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6"
Oct 30 01:25:13.162824 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:13.162800    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c52a67b5-78c3-4964-9f99-056cd1fd7a55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6\" (UID: \"c52a67b5-78c3-4964-9f99-056cd1fd7a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6"
Oct 30 01:25:13.164523 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:13.164504    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c52a67b5-78c3-4964-9f99-056cd1fd7a55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6\" (UID: \"c52a67b5-78c3-4964-9f99-056cd1fd7a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6"
Oct 30 01:25:13.185421 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:13.185393    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8p7k\" (UniqueName: \"kubernetes.io/projected/c52a67b5-78c3-4964-9f99-056cd1fd7a55-kube-api-access-f8p7k\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6\" (UID: \"c52a67b5-78c3-4964-9f99-056cd1fd7a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6"
Oct 30 01:25:13.298016 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:13.297988    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6"
Oct 30 01:25:13.598087 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:13.598058    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6"]
Oct 30 01:25:13.599340 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:25:13.599312    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52a67b5_78c3_4964_9f99_056cd1fd7a55.slice/crio-207685bd52bda34d0a013f7b121febb39622974dc2330e3c4a9896e91740c199 WatchSource:0}: Error finding container 207685bd52bda34d0a013f7b121febb39622974dc2330e3c4a9896e91740c199: Status 404 returned error can't find the container with id 207685bd52bda34d0a013f7b121febb39622974dc2330e3c4a9896e91740c199
Oct 30 01:25:14.260348 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:14.260269    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6" event={"ID":"c52a67b5-78c3-4964-9f99-056cd1fd7a55","Type":"ContainerStarted","Data":"77128293d73b3e8d364ae5d21b598853a47dbc12c286293ad33af07826b3aea8"}
Oct 30 01:25:14.260348 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:14.260316    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6" event={"ID":"c52a67b5-78c3-4964-9f99-056cd1fd7a55","Type":"ContainerStarted","Data":"207685bd52bda34d0a013f7b121febb39622974dc2330e3c4a9896e91740c199"}
Oct 30 01:25:14.298852 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:14.298807    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6" podStartSLOduration=2.298793912 podStartE2EDuration="2.298793912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:14.297313863 +0000 UTC m=+3008.154221618" watchObservedRunningTime="2026-10-30 01:25:14.298793912 +0000 UTC m=+3008.155701666"
Oct 30 01:25:15.266324 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:15.266280    6099 generic.go:356] "Generic (PLEG): container finished" podID="c52a67b5-78c3-4964-9f99-056cd1fd7a55" containerID="77128293d73b3e8d364ae5d21b598853a47dbc12c286293ad33af07826b3aea8" exitCode=0
Oct 30 01:25:15.266842 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:15.266334    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6" event={"ID":"c52a67b5-78c3-4964-9f99-056cd1fd7a55","Type":"ContainerDied","Data":"77128293d73b3e8d364ae5d21b598853a47dbc12c286293ad33af07826b3aea8"}
Oct 30 01:25:16.450742 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.450695    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6"
Oct 30 01:25:16.591217 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.591187    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:25:16.605638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.605617    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c52a67b5-78c3-4964-9f99-056cd1fd7a55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c52a67b5-78c3-4964-9f99-056cd1fd7a55-maas-api-openshift-ca\") pod \"c52a67b5-78c3-4964-9f99-056cd1fd7a55\" (UID: \"c52a67b5-78c3-4964-9f99-056cd1fd7a55\") "
Oct 30 01:25:16.605773 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.605724    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c52a67b5-78c3-4964-9f99-056cd1fd7a55-kube-api-access-f8p7k\" (UniqueName: \"kubernetes.io/projected/c52a67b5-78c3-4964-9f99-056cd1fd7a55-kube-api-access-f8p7k\") pod \"c52a67b5-78c3-4964-9f99-056cd1fd7a55\" (UID: \"c52a67b5-78c3-4964-9f99-056cd1fd7a55\") "
Oct 30 01:25:16.605773 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.605763    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c52a67b5-78c3-4964-9f99-056cd1fd7a55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c52a67b5-78c3-4964-9f99-056cd1fd7a55-maas-api-xks-ca\") pod \"c52a67b5-78c3-4964-9f99-056cd1fd7a55\" (UID: \"c52a67b5-78c3-4964-9f99-056cd1fd7a55\") "
Oct 30 01:25:16.606061 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.606036    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52a67b5-78c3-4964-9f99-056cd1fd7a55-maas-api-openshift-ca" pod "c52a67b5-78c3-4964-9f99-056cd1fd7a55" (UID: "c52a67b5-78c3-4964-9f99-056cd1fd7a55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:16.606253 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.606231    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c52a67b5-78c3-4964-9f99-056cd1fd7a55-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:16.607806 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.607781    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52a67b5-78c3-4964-9f99-056cd1fd7a55-kube-api-access-f8p7k" pod "c52a67b5-78c3-4964-9f99-056cd1fd7a55" (UID: "c52a67b5-78c3-4964-9f99-056cd1fd7a55"). InnerVolumeSpecName "kube-api-access-f8p7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:16.607907 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.607813    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c52a67b5-78c3-4964-9f99-056cd1fd7a55-maas-api-xks-ca" pod "c52a67b5-78c3-4964-9f99-056cd1fd7a55" (UID: "c52a67b5-78c3-4964-9f99-056cd1fd7a55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:16.629434 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:25:16.629407    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:25:16.707219 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.707188    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bbcb61d-3921-444a-bcc9-1b8382320f04-kube-api-access-4zjbs\" (UniqueName: \"kubernetes.io/projected/5bbcb61d-3921-444a-bcc9-1b8382320f04-kube-api-access-4zjbs\") pod \"5bbcb61d-3921-444a-bcc9-1b8382320f04\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") "
Oct 30 01:25:16.707419 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.707233    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-token\" (UniqueName: \"kubernetes.io/projected/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-token\") pod \"5bbcb61d-3921-444a-bcc9-1b8382320f04\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") "
Oct 30 01:25:16.707419 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.707284    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-envoy\") pod \"5bbcb61d-3921-444a-bcc9-1b8382320f04\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") "
Oct 30 01:25:16.707419 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.707338    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-credential-socket\") pod \"5bbcb61d-3921-444a-bcc9-1b8382320f04\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") "
Oct 30 01:25:16.707419 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.707371    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-data\") pod \"5bbcb61d-3921-444a-bcc9-1b8382320f04\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") "
Oct 30 01:25:16.707419 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.707417    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-workload-certs\") pod \"5bbcb61d-3921-444a-bcc9-1b8382320f04\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") "
Oct 30 01:25:16.707756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.707569    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-workload-socket\") pod \"5bbcb61d-3921-444a-bcc9-1b8382320f04\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") "
Oct 30 01:25:16.707756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.707600    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-podinfo\") pod \"5bbcb61d-3921-444a-bcc9-1b8382320f04\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") "
Oct 30 01:25:16.707756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.707658    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bbcb61d-3921-444a-bcc9-1b8382320f04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbcb61d-3921-444a-bcc9-1b8382320f04-istiod-ca-cert\") pod \"5bbcb61d-3921-444a-bcc9-1b8382320f04\" (UID: \"5bbcb61d-3921-444a-bcc9-1b8382320f04\") "
Oct 30 01:25:16.707756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.707698    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-data" pod "5bbcb61d-3921-444a-bcc9-1b8382320f04" (UID: "5bbcb61d-3921-444a-bcc9-1b8382320f04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:16.708012 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.707745    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-credential-socket" pod "5bbcb61d-3921-444a-bcc9-1b8382320f04" (UID: "5bbcb61d-3921-444a-bcc9-1b8382320f04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:16.708012 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.707847    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-workload-certs" pod "5bbcb61d-3921-444a-bcc9-1b8382320f04" (UID: "5bbcb61d-3921-444a-bcc9-1b8382320f04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:16.708012 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.707897    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-workload-socket" pod "5bbcb61d-3921-444a-bcc9-1b8382320f04" (UID: "5bbcb61d-3921-444a-bcc9-1b8382320f04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:16.708179 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.708163    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bbcb61d-3921-444a-bcc9-1b8382320f04-istiod-ca-cert" pod "5bbcb61d-3921-444a-bcc9-1b8382320f04" (UID: "5bbcb61d-3921-444a-bcc9-1b8382320f04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:16.708274 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.708254    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8p7k\" (UniqueName: \"kubernetes.io/projected/c52a67b5-78c3-4964-9f99-056cd1fd7a55-kube-api-access-f8p7k\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:16.708343 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.708279    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:16.708343 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.708296    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:16.708343 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.708311    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c52a67b5-78c3-4964-9f99-056cd1fd7a55-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:16.708343 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.708323    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:16.708343 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.708335    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:16.708631 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.708350    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbcb61d-3921-444a-bcc9-1b8382320f04-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:16.709881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.709858    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-envoy" pod "5bbcb61d-3921-444a-bcc9-1b8382320f04" (UID: "5bbcb61d-3921-444a-bcc9-1b8382320f04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:16.710123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.710105    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbcb61d-3921-444a-bcc9-1b8382320f04-kube-api-access-4zjbs" pod "5bbcb61d-3921-444a-bcc9-1b8382320f04" (UID: "5bbcb61d-3921-444a-bcc9-1b8382320f04"). InnerVolumeSpecName "kube-api-access-4zjbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:16.710211 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.710135    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-token" pod "5bbcb61d-3921-444a-bcc9-1b8382320f04" (UID: "5bbcb61d-3921-444a-bcc9-1b8382320f04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:16.710211 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.710168    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-podinfo" pod "5bbcb61d-3921-444a-bcc9-1b8382320f04" (UID: "5bbcb61d-3921-444a-bcc9-1b8382320f04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:25:16.809937 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.809905    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:16.809937 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.809935    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zjbs\" (UniqueName: \"kubernetes.io/projected/5bbcb61d-3921-444a-bcc9-1b8382320f04-kube-api-access-4zjbs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:16.810119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.809950    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:16.810119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:16.809962    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbcb61d-3921-444a-bcc9-1b8382320f04-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:17.280417 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:17.280381    6099 generic.go:356] "Generic (PLEG): container finished" podID="5bbcb61d-3921-444a-bcc9-1b8382320f04" containerID="2167109bfbe4f5c784b335dc1d0b085c551bb3191aa2e85d38052c0654af4d3a" exitCode=0
Oct 30 01:25:17.280620 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:17.280427    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"
Oct 30 01:25:17.280620 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:17.280469    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q" event={"ID":"5bbcb61d-3921-444a-bcc9-1b8382320f04","Type":"ContainerDied","Data":"2167109bfbe4f5c784b335dc1d0b085c551bb3191aa2e85d38052c0654af4d3a"}
Oct 30 01:25:17.280620 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:17.280526    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q" event={"ID":"5bbcb61d-3921-444a-bcc9-1b8382320f04","Type":"ContainerDied","Data":"1c7167ccd889d40e94487b8b534de8f6414478a7a41d96ab9f1656af34d5c87a"}
Oct 30 01:25:17.280620 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:17.280554    6099 scope.go:122] "RemoveContainer" containerID="2167109bfbe4f5c784b335dc1d0b085c551bb3191aa2e85d38052c0654af4d3a"
Oct 30 01:25:17.281930 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:17.281912    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6"
Oct 30 01:25:17.282068 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:17.281940    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6" event={"ID":"c52a67b5-78c3-4964-9f99-056cd1fd7a55","Type":"ContainerDied","Data":"207685bd52bda34d0a013f7b121febb39622974dc2330e3c4a9896e91740c199"}
Oct 30 01:25:17.282068 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:17.281980    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="207685bd52bda34d0a013f7b121febb39622974dc2330e3c4a9896e91740c199"
Oct 30 01:25:17.293056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:17.293036    6099 scope.go:122] "RemoveContainer" containerID="2167109bfbe4f5c784b335dc1d0b085c551bb3191aa2e85d38052c0654af4d3a"
Oct 30 01:25:17.293293 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:25:17.293277    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2167109bfbe4f5c784b335dc1d0b085c551bb3191aa2e85d38052c0654af4d3a\": container with ID starting with 2167109bfbe4f5c784b335dc1d0b085c551bb3191aa2e85d38052c0654af4d3a not found: ID does not exist" containerID="2167109bfbe4f5c784b335dc1d0b085c551bb3191aa2e85d38052c0654af4d3a"
Oct 30 01:25:17.293343 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:17.293300    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2167109bfbe4f5c784b335dc1d0b085c551bb3191aa2e85d38052c0654af4d3a"} err="failed to get container status \"2167109bfbe4f5c784b335dc1d0b085c551bb3191aa2e85d38052c0654af4d3a\": rpc error: code = NotFound desc = could not find container \"2167109bfbe4f5c784b335dc1d0b085c551bb3191aa2e85d38052c0654af4d3a\": container with ID starting with 2167109bfbe4f5c784b335dc1d0b085c551bb3191aa2e85d38052c0654af4d3a not found: ID does not exist"
Oct 30 01:25:17.364880 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:17.364852    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"]
Oct 30 01:25:17.369466 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:17.369437    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-95492e-openshift-default-7bd676f4b9-5sr8q"]
Oct 30 01:25:18.261582 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:18.261547    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bbcb61d-3921-444a-bcc9-1b8382320f04" path="/var/lib/kubelet/pods/5bbcb61d-3921-444a-bcc9-1b8382320f04/volumes"
Oct 30 01:25:23.395076 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:23.395038    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d"]
Oct 30 01:25:28.396376 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.396325    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d" podUID="e0d9984d-a336-4b7d-a72c-11a183059b3e" containerName="payload-processing" containerID="cri-o://0119e0d0d2f7926c607f7f1aa144236c90eceb006a20187e514443a8594a1fc6" gracePeriod=25
Oct 30 01:25:28.556233 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.556198    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"]
Oct 30 01:25:28.556475 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.556447    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb" podUID="7116e8fb-516c-486d-ae70-9f8cf1ad0d2a" containerName="maas-api" containerID="cri-o://91c4141c30a6dd725f849a88268adaf7c87bb51cf995ad61ac1c83d5827892dc" gracePeriod=30
Oct 30 01:25:28.752522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.752491    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d"
Oct 30 01:25:28.813415 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.813390    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"
Oct 30 01:25:28.834267 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.834241    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-openshift-service-ca\") pod \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\" (UID: \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\") "
Oct 30 01:25:28.834406 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.834278    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-kube-api-access-crxzn\" (UniqueName: \"kubernetes.io/projected/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-kube-api-access-crxzn\") pod \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\" (UID: \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\") "
Oct 30 01:25:28.834406 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.834323    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d9984d-a336-4b7d-a72c-11a183059b3e-kube-api-access-xzr82\" (UniqueName: \"kubernetes.io/projected/e0d9984d-a336-4b7d-a72c-11a183059b3e-kube-api-access-xzr82\") pod \"e0d9984d-a336-4b7d-a72c-11a183059b3e\" (UID: \"e0d9984d-a336-4b7d-a72c-11a183059b3e\") "
Oct 30 01:25:28.834544 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.834469    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d9984d-a336-4b7d-a72c-11a183059b3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d9984d-a336-4b7d-a72c-11a183059b3e-plugins-config-volume\") pod \"e0d9984d-a336-4b7d-a72c-11a183059b3e\" (UID: \"e0d9984d-a336-4b7d-a72c-11a183059b3e\") "
Oct 30 01:25:28.834609 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.834557    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-metrics-certs\") pod \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\" (UID: \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\") "
Oct 30 01:25:28.834609 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.834597    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-maas-api-tls\") pod \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\" (UID: \"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a\") "
Oct 30 01:25:28.834747 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.834611    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-openshift-service-ca" pod "7116e8fb-516c-486d-ae70-9f8cf1ad0d2a" (UID: "7116e8fb-516c-486d-ae70-9f8cf1ad0d2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:28.834897 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.834877    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d9984d-a336-4b7d-a72c-11a183059b3e-plugins-config-volume" pod "e0d9984d-a336-4b7d-a72c-11a183059b3e" (UID: "e0d9984d-a336-4b7d-a72c-11a183059b3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:28.835107 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.835066    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:28.835107 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.835099    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d9984d-a336-4b7d-a72c-11a183059b3e-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:28.836622 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.836597    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d9984d-a336-4b7d-a72c-11a183059b3e-kube-api-access-xzr82" pod "e0d9984d-a336-4b7d-a72c-11a183059b3e" (UID: "e0d9984d-a336-4b7d-a72c-11a183059b3e"). InnerVolumeSpecName "kube-api-access-xzr82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:28.836748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.836617    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-kube-api-access-crxzn" pod "7116e8fb-516c-486d-ae70-9f8cf1ad0d2a" (UID: "7116e8fb-516c-486d-ae70-9f8cf1ad0d2a"). InnerVolumeSpecName "kube-api-access-crxzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:28.836748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.836644    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-metrics-certs" pod "7116e8fb-516c-486d-ae70-9f8cf1ad0d2a" (UID: "7116e8fb-516c-486d-ae70-9f8cf1ad0d2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:28.836748 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.836665    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-maas-api-tls" pod "7116e8fb-516c-486d-ae70-9f8cf1ad0d2a" (UID: "7116e8fb-516c-486d-ae70-9f8cf1ad0d2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:28.936558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.936481    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crxzn\" (UniqueName: \"kubernetes.io/projected/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-kube-api-access-crxzn\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:28.936558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.936507    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzr82\" (UniqueName: \"kubernetes.io/projected/e0d9984d-a336-4b7d-a72c-11a183059b3e-kube-api-access-xzr82\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:28.936558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.936517    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:28.936558 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:28.936527    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:29.356212 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.356176    6099 generic.go:356] "Generic (PLEG): container finished" podID="e0d9984d-a336-4b7d-a72c-11a183059b3e" containerID="0119e0d0d2f7926c607f7f1aa144236c90eceb006a20187e514443a8594a1fc6" exitCode=0
Oct 30 01:25:29.356427 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.356240    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d"
Oct 30 01:25:29.356427 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.356275    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d" event={"ID":"e0d9984d-a336-4b7d-a72c-11a183059b3e","Type":"ContainerDied","Data":"0119e0d0d2f7926c607f7f1aa144236c90eceb006a20187e514443a8594a1fc6"}
Oct 30 01:25:29.356427 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.356343    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d" event={"ID":"e0d9984d-a336-4b7d-a72c-11a183059b3e","Type":"ContainerDied","Data":"f8510269b69a2c88d9fe199e8664276cc047a4f5e5bed64854abb726d2cfeb0d"}
Oct 30 01:25:29.356427 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.356375    6099 scope.go:122] "RemoveContainer" containerID="0119e0d0d2f7926c607f7f1aa144236c90eceb006a20187e514443a8594a1fc6"
Oct 30 01:25:29.357770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.357749    6099 generic.go:356] "Generic (PLEG): container finished" podID="7116e8fb-516c-486d-ae70-9f8cf1ad0d2a" containerID="91c4141c30a6dd725f849a88268adaf7c87bb51cf995ad61ac1c83d5827892dc" exitCode=0
Oct 30 01:25:29.357865 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.357775    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb" event={"ID":"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a","Type":"ContainerDied","Data":"91c4141c30a6dd725f849a88268adaf7c87bb51cf995ad61ac1c83d5827892dc"}
Oct 30 01:25:29.357865 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.357791    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"
Oct 30 01:25:29.357865 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.357823    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb" event={"ID":"7116e8fb-516c-486d-ae70-9f8cf1ad0d2a","Type":"ContainerDied","Data":"c26c049674f74c70431bafbcf6f51e26a5e133b92d2d6548e8422e44b85a5d45"}
Oct 30 01:25:29.367775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.367582    6099 scope.go:122] "RemoveContainer" containerID="0119e0d0d2f7926c607f7f1aa144236c90eceb006a20187e514443a8594a1fc6"
Oct 30 01:25:29.368036 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:25:29.368012    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0119e0d0d2f7926c607f7f1aa144236c90eceb006a20187e514443a8594a1fc6\": container with ID starting with 0119e0d0d2f7926c607f7f1aa144236c90eceb006a20187e514443a8594a1fc6 not found: ID does not exist" containerID="0119e0d0d2f7926c607f7f1aa144236c90eceb006a20187e514443a8594a1fc6"
Oct 30 01:25:29.368124 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.368050    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0119e0d0d2f7926c607f7f1aa144236c90eceb006a20187e514443a8594a1fc6"} err="failed to get container status \"0119e0d0d2f7926c607f7f1aa144236c90eceb006a20187e514443a8594a1fc6\": rpc error: code = NotFound desc = could not find container \"0119e0d0d2f7926c607f7f1aa144236c90eceb006a20187e514443a8594a1fc6\": container with ID starting with 0119e0d0d2f7926c607f7f1aa144236c90eceb006a20187e514443a8594a1fc6 not found: ID does not exist"
Oct 30 01:25:29.368124 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.368076    6099 scope.go:122] "RemoveContainer" containerID="91c4141c30a6dd725f849a88268adaf7c87bb51cf995ad61ac1c83d5827892dc"
Oct 30 01:25:29.377822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.377785    6099 scope.go:122] "RemoveContainer" containerID="91c4141c30a6dd725f849a88268adaf7c87bb51cf995ad61ac1c83d5827892dc"
Oct 30 01:25:29.378058 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:25:29.378040    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c4141c30a6dd725f849a88268adaf7c87bb51cf995ad61ac1c83d5827892dc\": container with ID starting with 91c4141c30a6dd725f849a88268adaf7c87bb51cf995ad61ac1c83d5827892dc not found: ID does not exist" containerID="91c4141c30a6dd725f849a88268adaf7c87bb51cf995ad61ac1c83d5827892dc"
Oct 30 01:25:29.378117 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.378076    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c4141c30a6dd725f849a88268adaf7c87bb51cf995ad61ac1c83d5827892dc"} err="failed to get container status \"91c4141c30a6dd725f849a88268adaf7c87bb51cf995ad61ac1c83d5827892dc\": rpc error: code = NotFound desc = could not find container \"91c4141c30a6dd725f849a88268adaf7c87bb51cf995ad61ac1c83d5827892dc\": container with ID starting with 91c4141c30a6dd725f849a88268adaf7c87bb51cf995ad61ac1c83d5827892dc not found: ID does not exist"
Oct 30 01:25:29.406753 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.406727    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d"]
Oct 30 01:25:29.412654 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.412631    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-d01638-5668596858-chl8d"]
Oct 30 01:25:29.433930 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.433907    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6"]
Oct 30 01:25:29.436315 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.436296    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d01638-jxkp6"]
Oct 30 01:25:29.445483 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.445462    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"]
Oct 30 01:25:29.449836 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:29.449816    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d01638-7dc64596f6-v2nmb"]
Oct 30 01:25:30.260753 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:30.260692    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7116e8fb-516c-486d-ae70-9f8cf1ad0d2a" path="/var/lib/kubelet/pods/7116e8fb-516c-486d-ae70-9f8cf1ad0d2a/volumes"
Oct 30 01:25:30.261178 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:30.261163    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c52a67b5-78c3-4964-9f99-056cd1fd7a55" path="/var/lib/kubelet/pods/c52a67b5-78c3-4964-9f99-056cd1fd7a55/volumes"
Oct 30 01:25:30.261535 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:30.261521    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d9984d-a336-4b7d-a72c-11a183059b3e" path="/var/lib/kubelet/pods/e0d9984d-a336-4b7d-a72c-11a183059b3e/volumes"
Oct 30 01:25:32.835739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:32.835692    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:25:36.072871 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.072836    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"]
Oct 30 01:25:36.073382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.073145    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62" podUID="3e0b7e93-d524-4443-98ca-a0993b9232c0" containerName="istio-proxy" containerID="cri-o://ad59b0471e3cce4ece2a3bc1e68f215f6d08b2bef3920668a883439a0c2a4b58" gracePeriod=30
Oct 30 01:25:36.083680 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.083655    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"]
Oct 30 01:25:36.083904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.083865    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb" podUID="227245db-8ac3-47c1-ae8c-ba5ff96202c6" containerName="istio-proxy" containerID="cri-o://06f34b48462002ac3e7cb3748338d0bd297ec9a09a1ac0eba95d515a2510f330" gracePeriod=30
Oct 30 01:25:36.086624 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.086575    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"]
Oct 30 01:25:36.086830 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.086805    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq" podUID="f26062a6-cf7a-4faf-a955-5e408bc47c08" containerName="istio-proxy" containerID="cri-o://93d5c66001633e84d2952abc81e3193a7f97d5f060bee02e676d8b62ecc0481c" gracePeriod=30
Oct 30 01:25:36.770995 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.770961    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88"]
Oct 30 01:25:36.771451 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.771437    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e0d9984d-a336-4b7d-a72c-11a183059b3e" containerName="payload-processing"
Oct 30 01:25:36.771499 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.771452    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d9984d-a336-4b7d-a72c-11a183059b3e" containerName="payload-processing"
Oct 30 01:25:36.771499 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.771476    6099 cpu_manager.go:414] "containerMap: removing container" podUID="c52a67b5-78c3-4964-9f99-056cd1fd7a55" containerName="revoke-keys"
Oct 30 01:25:36.771499 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.771481    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52a67b5-78c3-4964-9f99-056cd1fd7a55" containerName="revoke-keys"
Oct 30 01:25:36.771602 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.771503    6099 cpu_manager.go:414] "containerMap: removing container" podUID="7116e8fb-516c-486d-ae70-9f8cf1ad0d2a" containerName="maas-api"
Oct 30 01:25:36.771602 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.771509    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7116e8fb-516c-486d-ae70-9f8cf1ad0d2a" containerName="maas-api"
Oct 30 01:25:36.771602 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.771523    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5bbcb61d-3921-444a-bcc9-1b8382320f04" containerName="istio-proxy"
Oct 30 01:25:36.771602 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.771527    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bbcb61d-3921-444a-bcc9-1b8382320f04" containerName="istio-proxy"
Oct 30 01:25:36.771602 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.771599    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7116e8fb-516c-486d-ae70-9f8cf1ad0d2a" containerName="maas-api"
Oct 30 01:25:36.771800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.771611    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bbcb61d-3921-444a-bcc9-1b8382320f04" containerName="istio-proxy"
Oct 30 01:25:36.771800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.771621    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d9984d-a336-4b7d-a72c-11a183059b3e" containerName="payload-processing"
Oct 30 01:25:36.771800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.771628    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52a67b5-78c3-4964-9f99-056cd1fd7a55" containerName="revoke-keys"
Oct 30 01:25:36.776452 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.776433    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88"
Oct 30 01:25:36.792057 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.792032    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:25:36.795163 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.795140    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88"]
Oct 30 01:25:36.825588 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.825561    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79665e1a-c51c-4623-8c73-f505c73bf38b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88\" (UID: \"79665e1a-c51c-4623-8c73-f505c73bf38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88"
Oct 30 01:25:36.825780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.825689    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79665e1a-c51c-4623-8c73-f505c73bf38b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88\" (UID: \"79665e1a-c51c-4623-8c73-f505c73bf38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88"
Oct 30 01:25:36.825867 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.825799    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77dnc\" (UniqueName: \"kubernetes.io/projected/79665e1a-c51c-4623-8c73-f505c73bf38b-kube-api-access-77dnc\") pod \"maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88\" (UID: \"79665e1a-c51c-4623-8c73-f505c73bf38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88"
Oct 30 01:25:36.885727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.885673    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:25:36.927456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.927425    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79665e1a-c51c-4623-8c73-f505c73bf38b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88\" (UID: \"79665e1a-c51c-4623-8c73-f505c73bf38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88"
Oct 30 01:25:36.927612 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.927507    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79665e1a-c51c-4623-8c73-f505c73bf38b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88\" (UID: \"79665e1a-c51c-4623-8c73-f505c73bf38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88"
Oct 30 01:25:36.927612 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.927542    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77dnc\" (UniqueName: \"kubernetes.io/projected/79665e1a-c51c-4623-8c73-f505c73bf38b-kube-api-access-77dnc\") pod \"maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88\" (UID: \"79665e1a-c51c-4623-8c73-f505c73bf38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88"
Oct 30 01:25:36.928191 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.928166    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79665e1a-c51c-4623-8c73-f505c73bf38b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88\" (UID: \"79665e1a-c51c-4623-8c73-f505c73bf38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88"
Oct 30 01:25:36.929933 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.929911    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79665e1a-c51c-4623-8c73-f505c73bf38b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88\" (UID: \"79665e1a-c51c-4623-8c73-f505c73bf38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88"
Oct 30 01:25:36.953128 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:36.953102    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77dnc\" (UniqueName: \"kubernetes.io/projected/79665e1a-c51c-4623-8c73-f505c73bf38b-kube-api-access-77dnc\") pod \"maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88\" (UID: \"79665e1a-c51c-4623-8c73-f505c73bf38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88"
Oct 30 01:25:37.090482 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:37.090451    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88"
Oct 30 01:25:37.399792 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:37.399766    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88"]
Oct 30 01:25:37.400379 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:25:37.400356    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79665e1a_c51c_4623_8c73_f505c73bf38b.slice/crio-10a4c19ffe48d74707711ee2ff4349e350bf08af88f7cfe9c295a564583ceb9c WatchSource:0}: Error finding container 10a4c19ffe48d74707711ee2ff4349e350bf08af88f7cfe9c295a564583ceb9c: Status 404 returned error can't find the container with id 10a4c19ffe48d74707711ee2ff4349e350bf08af88f7cfe9c295a564583ceb9c
Oct 30 01:25:37.409069 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:37.409042    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88" event={"ID":"79665e1a-c51c-4623-8c73-f505c73bf38b","Type":"ContainerStarted","Data":"10a4c19ffe48d74707711ee2ff4349e350bf08af88f7cfe9c295a564583ceb9c"}
Oct 30 01:25:38.416260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:38.416226    6099 generic.go:356] "Generic (PLEG): container finished" podID="79665e1a-c51c-4623-8c73-f505c73bf38b" containerID="abcefa3373930b9c6520a20aaa3c2157a7d102ad4a1b13c7566c9367d8024489" exitCode=0
Oct 30 01:25:38.416590 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:38.416310    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88" event={"ID":"79665e1a-c51c-4623-8c73-f505c73bf38b","Type":"ContainerDied","Data":"abcefa3373930b9c6520a20aaa3c2157a7d102ad4a1b13c7566c9367d8024489"}
Oct 30 01:25:39.554176 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:39.554153    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88"
Oct 30 01:25:39.658764 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:39.658729    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79665e1a-c51c-4623-8c73-f505c73bf38b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79665e1a-c51c-4623-8c73-f505c73bf38b-maas-api-openshift-ca\") pod \"79665e1a-c51c-4623-8c73-f505c73bf38b\" (UID: \"79665e1a-c51c-4623-8c73-f505c73bf38b\") "
Oct 30 01:25:39.658950 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:39.658832    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79665e1a-c51c-4623-8c73-f505c73bf38b-kube-api-access-77dnc\" (UniqueName: \"kubernetes.io/projected/79665e1a-c51c-4623-8c73-f505c73bf38b-kube-api-access-77dnc\") pod \"79665e1a-c51c-4623-8c73-f505c73bf38b\" (UID: \"79665e1a-c51c-4623-8c73-f505c73bf38b\") "
Oct 30 01:25:39.659038 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:39.659004    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79665e1a-c51c-4623-8c73-f505c73bf38b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79665e1a-c51c-4623-8c73-f505c73bf38b-maas-api-xks-ca\") pod \"79665e1a-c51c-4623-8c73-f505c73bf38b\" (UID: \"79665e1a-c51c-4623-8c73-f505c73bf38b\") "
Oct 30 01:25:39.659100 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:39.659057    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79665e1a-c51c-4623-8c73-f505c73bf38b-maas-api-openshift-ca" pod "79665e1a-c51c-4623-8c73-f505c73bf38b" (UID: "79665e1a-c51c-4623-8c73-f505c73bf38b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:39.659409 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:39.659390    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79665e1a-c51c-4623-8c73-f505c73bf38b-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:39.661020 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:39.660981    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79665e1a-c51c-4623-8c73-f505c73bf38b-kube-api-access-77dnc" pod "79665e1a-c51c-4623-8c73-f505c73bf38b" (UID: "79665e1a-c51c-4623-8c73-f505c73bf38b"). InnerVolumeSpecName "kube-api-access-77dnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:39.661020 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:39.660999    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79665e1a-c51c-4623-8c73-f505c73bf38b-maas-api-xks-ca" pod "79665e1a-c51c-4623-8c73-f505c73bf38b" (UID: "79665e1a-c51c-4623-8c73-f505c73bf38b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:39.760597 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:39.760516    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77dnc\" (UniqueName: \"kubernetes.io/projected/79665e1a-c51c-4623-8c73-f505c73bf38b-kube-api-access-77dnc\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:39.760597 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:39.760553    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79665e1a-c51c-4623-8c73-f505c73bf38b-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:40.429075 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:40.429043    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88"
Oct 30 01:25:40.429252 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:40.429072    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88" event={"ID":"79665e1a-c51c-4623-8c73-f505c73bf38b","Type":"ContainerDied","Data":"10a4c19ffe48d74707711ee2ff4349e350bf08af88f7cfe9c295a564583ceb9c"}
Oct 30 01:25:40.429252 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:40.429123    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10a4c19ffe48d74707711ee2ff4349e350bf08af88f7cfe9c295a564583ceb9c"
Oct 30 01:25:41.363357 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.363328    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:25:41.386649 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.386567    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:25:41.395065 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.395038    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:25:41.438958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.438926    6099 generic.go:356] "Generic (PLEG): container finished" podID="227245db-8ac3-47c1-ae8c-ba5ff96202c6" containerID="06f34b48462002ac3e7cb3748338d0bd297ec9a09a1ac0eba95d515a2510f330" exitCode=0
Oct 30 01:25:41.439125 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.438973    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"
Oct 30 01:25:41.439125 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.439010    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb" event={"ID":"227245db-8ac3-47c1-ae8c-ba5ff96202c6","Type":"ContainerDied","Data":"06f34b48462002ac3e7cb3748338d0bd297ec9a09a1ac0eba95d515a2510f330"}
Oct 30 01:25:41.439125 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.439082    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb" event={"ID":"227245db-8ac3-47c1-ae8c-ba5ff96202c6","Type":"ContainerDied","Data":"a5b266f020b1b7679d2ffc43bd61b1b72b757e4c8b84210eb45c495f7c288e64"}
Oct 30 01:25:41.439125 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.439117    6099 scope.go:122] "RemoveContainer" containerID="06f34b48462002ac3e7cb3748338d0bd297ec9a09a1ac0eba95d515a2510f330"
Oct 30 01:25:41.440818 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.440685    6099 generic.go:356] "Generic (PLEG): container finished" podID="3e0b7e93-d524-4443-98ca-a0993b9232c0" containerID="ad59b0471e3cce4ece2a3bc1e68f215f6d08b2bef3920668a883439a0c2a4b58" exitCode=0
Oct 30 01:25:41.440818 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.440772    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"
Oct 30 01:25:41.441017 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.440936    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62" event={"ID":"3e0b7e93-d524-4443-98ca-a0993b9232c0","Type":"ContainerDied","Data":"ad59b0471e3cce4ece2a3bc1e68f215f6d08b2bef3920668a883439a0c2a4b58"}
Oct 30 01:25:41.441017 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.440999    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62" event={"ID":"3e0b7e93-d524-4443-98ca-a0993b9232c0","Type":"ContainerDied","Data":"d89c4d02e6e7a22dab5c771add1b173ff5fac1447fa9668eaeea56e3d576b0a1"}
Oct 30 01:25:41.442752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.442728    6099 generic.go:356] "Generic (PLEG): container finished" podID="f26062a6-cf7a-4faf-a955-5e408bc47c08" containerID="93d5c66001633e84d2952abc81e3193a7f97d5f060bee02e676d8b62ecc0481c" exitCode=0
Oct 30 01:25:41.442880 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.442780    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq" event={"ID":"f26062a6-cf7a-4faf-a955-5e408bc47c08","Type":"ContainerDied","Data":"93d5c66001633e84d2952abc81e3193a7f97d5f060bee02e676d8b62ecc0481c"}
Oct 30 01:25:41.442880 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.442819    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq" event={"ID":"f26062a6-cf7a-4faf-a955-5e408bc47c08","Type":"ContainerDied","Data":"207d9c1179a96db2e71751c9de5c5755775905c610606fee9eee435d03b269d9"}
Oct 30 01:25:41.443013 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.442880    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"
Oct 30 01:25:41.461074 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.461050    6099 scope.go:122] "RemoveContainer" containerID="06f34b48462002ac3e7cb3748338d0bd297ec9a09a1ac0eba95d515a2510f330"
Oct 30 01:25:41.463018 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:25:41.462363    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06f34b48462002ac3e7cb3748338d0bd297ec9a09a1ac0eba95d515a2510f330\": container with ID starting with 06f34b48462002ac3e7cb3748338d0bd297ec9a09a1ac0eba95d515a2510f330 not found: ID does not exist" containerID="06f34b48462002ac3e7cb3748338d0bd297ec9a09a1ac0eba95d515a2510f330"
Oct 30 01:25:41.463018 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.462418    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06f34b48462002ac3e7cb3748338d0bd297ec9a09a1ac0eba95d515a2510f330"} err="failed to get container status \"06f34b48462002ac3e7cb3748338d0bd297ec9a09a1ac0eba95d515a2510f330\": rpc error: code = NotFound desc = could not find container \"06f34b48462002ac3e7cb3748338d0bd297ec9a09a1ac0eba95d515a2510f330\": container with ID starting with 06f34b48462002ac3e7cb3748338d0bd297ec9a09a1ac0eba95d515a2510f330 not found: ID does not exist"
Oct 30 01:25:41.463018 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.462444    6099 scope.go:122] "RemoveContainer" containerID="ad59b0471e3cce4ece2a3bc1e68f215f6d08b2bef3920668a883439a0c2a4b58"
Oct 30 01:25:41.478311 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.478283    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-envoy\") pod \"f26062a6-cf7a-4faf-a955-5e408bc47c08\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") "
Oct 30 01:25:41.478439 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.478342    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26062a6-cf7a-4faf-a955-5e408bc47c08-kube-api-access-9bh8n\" (UniqueName: \"kubernetes.io/projected/f26062a6-cf7a-4faf-a955-5e408bc47c08-kube-api-access-9bh8n\") pod \"f26062a6-cf7a-4faf-a955-5e408bc47c08\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") "
Oct 30 01:25:41.478439 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.478373    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-workload-certs\") pod \"3e0b7e93-d524-4443-98ca-a0993b9232c0\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") "
Oct 30 01:25:41.478439 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.478413    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-workload-socket\") pod \"f26062a6-cf7a-4faf-a955-5e408bc47c08\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") "
Oct 30 01:25:41.478626 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.478600    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/227245db-8ac3-47c1-ae8c-ba5ff96202c6-kube-api-access-kxh96\" (UniqueName: \"kubernetes.io/projected/227245db-8ac3-47c1-ae8c-ba5ff96202c6-kube-api-access-kxh96\") pod \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") "
Oct 30 01:25:41.478693 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.478649    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-data\") pod \"3e0b7e93-d524-4443-98ca-a0993b9232c0\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") "
Oct 30 01:25:41.478693 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.478653    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-workload-certs" pod "3e0b7e93-d524-4443-98ca-a0993b9232c0" (UID: "3e0b7e93-d524-4443-98ca-a0993b9232c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:41.478831 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.478699    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-workload-socket\") pod \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") "
Oct 30 01:25:41.478831 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.478743    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-token\" (UniqueName: \"kubernetes.io/projected/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-token\") pod \"3e0b7e93-d524-4443-98ca-a0993b9232c0\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") "
Oct 30 01:25:41.478831 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.478775    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-podinfo\") pod \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") "
Oct 30 01:25:41.478831 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.478774    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-workload-socket" pod "f26062a6-cf7a-4faf-a955-5e408bc47c08" (UID: "f26062a6-cf7a-4faf-a955-5e408bc47c08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:41.478831 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.478812    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e0b7e93-d524-4443-98ca-a0993b9232c0-kube-api-access-mw8k8\" (UniqueName: \"kubernetes.io/projected/3e0b7e93-d524-4443-98ca-a0993b9232c0-kube-api-access-mw8k8\") pod \"3e0b7e93-d524-4443-98ca-a0993b9232c0\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") "
Oct 30 01:25:41.479124 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.478888    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-credential-socket\") pod \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") "
Oct 30 01:25:41.479124 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.478904    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-data" pod "3e0b7e93-d524-4443-98ca-a0993b9232c0" (UID: "3e0b7e93-d524-4443-98ca-a0993b9232c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:41.479124 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.478926    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-podinfo\") pod \"3e0b7e93-d524-4443-98ca-a0993b9232c0\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") "
Oct 30 01:25:41.479124 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.478959    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-token\" (UniqueName: \"kubernetes.io/projected/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-token\") pod \"f26062a6-cf7a-4faf-a955-5e408bc47c08\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") "
Oct 30 01:25:41.479124 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.479036    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istiod-ca-cert\") pod \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") "
Oct 30 01:25:41.479124 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.479078    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e0b7e93-d524-4443-98ca-a0993b9232c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e0b7e93-d524-4443-98ca-a0993b9232c0-istiod-ca-cert\") pod \"3e0b7e93-d524-4443-98ca-a0993b9232c0\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") "
Oct 30 01:25:41.479124 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.479110    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f26062a6-cf7a-4faf-a955-5e408bc47c08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26062a6-cf7a-4faf-a955-5e408bc47c08-istiod-ca-cert\") pod \"f26062a6-cf7a-4faf-a955-5e408bc47c08\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") "
Oct 30 01:25:41.479497 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.479157    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-credential-socket\") pod \"f26062a6-cf7a-4faf-a955-5e408bc47c08\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") "
Oct 30 01:25:41.479497 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.479220    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-credential-socket\") pod \"3e0b7e93-d524-4443-98ca-a0993b9232c0\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") "
Oct 30 01:25:41.479497 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.479254    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-podinfo\") pod \"f26062a6-cf7a-4faf-a955-5e408bc47c08\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") "
Oct 30 01:25:41.479497 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.479283    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-envoy\") pod \"3e0b7e93-d524-4443-98ca-a0993b9232c0\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") "
Oct 30 01:25:41.479497 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.479323    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-workload-socket\") pod \"3e0b7e93-d524-4443-98ca-a0993b9232c0\" (UID: \"3e0b7e93-d524-4443-98ca-a0993b9232c0\") "
Oct 30 01:25:41.479497 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.479379    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-token\" (UniqueName: \"kubernetes.io/projected/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-token\") pod \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") "
Oct 30 01:25:41.479497 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.479412    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-workload-certs\") pod \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") "
Oct 30 01:25:41.482739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.482219    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-credential-socket" pod "227245db-8ac3-47c1-ae8c-ba5ff96202c6" (UID: "227245db-8ac3-47c1-ae8c-ba5ff96202c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:41.482739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.482306    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227245db-8ac3-47c1-ae8c-ba5ff96202c6-kube-api-access-kxh96" pod "227245db-8ac3-47c1-ae8c-ba5ff96202c6" (UID: "227245db-8ac3-47c1-ae8c-ba5ff96202c6"). InnerVolumeSpecName "kube-api-access-kxh96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:41.482739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.482346    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-podinfo" pod "227245db-8ac3-47c1-ae8c-ba5ff96202c6" (UID: "227245db-8ac3-47c1-ae8c-ba5ff96202c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:25:41.482739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.482397    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-workload-socket" pod "227245db-8ac3-47c1-ae8c-ba5ff96202c6" (UID: "227245db-8ac3-47c1-ae8c-ba5ff96202c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:41.482739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.482427    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-envoy" pod "f26062a6-cf7a-4faf-a955-5e408bc47c08" (UID: "f26062a6-cf7a-4faf-a955-5e408bc47c08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:41.482739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.482615    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-credential-socket" pod "3e0b7e93-d524-4443-98ca-a0993b9232c0" (UID: "3e0b7e93-d524-4443-98ca-a0993b9232c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:41.482739 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.482736    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e0b7e93-d524-4443-98ca-a0993b9232c0-istiod-ca-cert" pod "3e0b7e93-d524-4443-98ca-a0993b9232c0" (UID: "3e0b7e93-d524-4443-98ca-a0993b9232c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:41.483167 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.482947    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f26062a6-cf7a-4faf-a955-5e408bc47c08-istiod-ca-cert" pod "f26062a6-cf7a-4faf-a955-5e408bc47c08" (UID: "f26062a6-cf7a-4faf-a955-5e408bc47c08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:41.483167 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.483094    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istiod-ca-cert" pod "227245db-8ac3-47c1-ae8c-ba5ff96202c6" (UID: "227245db-8ac3-47c1-ae8c-ba5ff96202c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:41.483273 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.483203    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-credential-socket" pod "f26062a6-cf7a-4faf-a955-5e408bc47c08" (UID: "f26062a6-cf7a-4faf-a955-5e408bc47c08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:41.484443 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.484319    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-podinfo" pod "3e0b7e93-d524-4443-98ca-a0993b9232c0" (UID: "3e0b7e93-d524-4443-98ca-a0993b9232c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:25:41.484957 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.484665    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-workload-certs" pod "227245db-8ac3-47c1-ae8c-ba5ff96202c6" (UID: "227245db-8ac3-47c1-ae8c-ba5ff96202c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:41.484957 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.484791    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-token" pod "3e0b7e93-d524-4443-98ca-a0993b9232c0" (UID: "3e0b7e93-d524-4443-98ca-a0993b9232c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:41.484957 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.484860    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e0b7e93-d524-4443-98ca-a0993b9232c0-kube-api-access-mw8k8" pod "3e0b7e93-d524-4443-98ca-a0993b9232c0" (UID: "3e0b7e93-d524-4443-98ca-a0993b9232c0"). InnerVolumeSpecName "kube-api-access-mw8k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:41.485190 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.484944    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-token" pod "f26062a6-cf7a-4faf-a955-5e408bc47c08" (UID: "f26062a6-cf7a-4faf-a955-5e408bc47c08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:41.485190 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.485027    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-envoy\") pod \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") "
Oct 30 01:25:41.485190 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.485076    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-workload-certs\") pod \"f26062a6-cf7a-4faf-a955-5e408bc47c08\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") "
Oct 30 01:25:41.485190 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.485126    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-data\") pod \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\" (UID: \"227245db-8ac3-47c1-ae8c-ba5ff96202c6\") "
Oct 30 01:25:41.485190 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.485187    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-data\") pod \"f26062a6-cf7a-4faf-a955-5e408bc47c08\" (UID: \"f26062a6-cf7a-4faf-a955-5e408bc47c08\") "
Oct 30 01:25:41.485515 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.485454    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-workload-certs" pod "f26062a6-cf7a-4faf-a955-5e408bc47c08" (UID: "f26062a6-cf7a-4faf-a955-5e408bc47c08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:41.486182 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486156    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e0b7e93-d524-4443-98ca-a0993b9232c0-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.486287 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486188    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26062a6-cf7a-4faf-a955-5e408bc47c08-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.486287 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486205    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.486287 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486230    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.486454 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486246    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.486454 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486377    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.486454 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486391    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.486454 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486410    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.486454 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486424    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.486454 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486439    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxh96\" (UniqueName: \"kubernetes.io/projected/227245db-8ac3-47c1-ae8c-ba5ff96202c6-kube-api-access-kxh96\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.486939 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486456    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.486939 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486476    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.486939 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486596    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.486939 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486610    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.486939 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486623    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw8k8\" (UniqueName: \"kubernetes.io/projected/3e0b7e93-d524-4443-98ca-a0993b9232c0-kube-api-access-mw8k8\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.486939 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486644    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.486939 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486659    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.486939 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486672    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.486939 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.486693    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.487405 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.487179    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-data" pod "f26062a6-cf7a-4faf-a955-5e408bc47c08" (UID: "f26062a6-cf7a-4faf-a955-5e408bc47c08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:41.492220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.487685    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-envoy" pod "227245db-8ac3-47c1-ae8c-ba5ff96202c6" (UID: "227245db-8ac3-47c1-ae8c-ba5ff96202c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:41.492220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.487759    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-data" pod "227245db-8ac3-47c1-ae8c-ba5ff96202c6" (UID: "227245db-8ac3-47c1-ae8c-ba5ff96202c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:41.492220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.487889    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-workload-socket" pod "3e0b7e93-d524-4443-98ca-a0993b9232c0" (UID: "3e0b7e93-d524-4443-98ca-a0993b9232c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:41.492220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.487981    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26062a6-cf7a-4faf-a955-5e408bc47c08-kube-api-access-9bh8n" pod "f26062a6-cf7a-4faf-a955-5e408bc47c08" (UID: "f26062a6-cf7a-4faf-a955-5e408bc47c08"). InnerVolumeSpecName "kube-api-access-9bh8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:41.492220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.488063    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-podinfo" pod "f26062a6-cf7a-4faf-a955-5e408bc47c08" (UID: "f26062a6-cf7a-4faf-a955-5e408bc47c08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:25:41.492220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.489534    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-envoy" pod "3e0b7e93-d524-4443-98ca-a0993b9232c0" (UID: "3e0b7e93-d524-4443-98ca-a0993b9232c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:41.492586 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.492344    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-token" pod "227245db-8ac3-47c1-ae8c-ba5ff96202c6" (UID: "227245db-8ac3-47c1-ae8c-ba5ff96202c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:41.507958 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.507938    6099 scope.go:122] "RemoveContainer" containerID="ad59b0471e3cce4ece2a3bc1e68f215f6d08b2bef3920668a883439a0c2a4b58"
Oct 30 01:25:41.508304 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:25:41.508284    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad59b0471e3cce4ece2a3bc1e68f215f6d08b2bef3920668a883439a0c2a4b58\": container with ID starting with ad59b0471e3cce4ece2a3bc1e68f215f6d08b2bef3920668a883439a0c2a4b58 not found: ID does not exist" containerID="ad59b0471e3cce4ece2a3bc1e68f215f6d08b2bef3920668a883439a0c2a4b58"
Oct 30 01:25:41.508371 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.508312    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad59b0471e3cce4ece2a3bc1e68f215f6d08b2bef3920668a883439a0c2a4b58"} err="failed to get container status \"ad59b0471e3cce4ece2a3bc1e68f215f6d08b2bef3920668a883439a0c2a4b58\": rpc error: code = NotFound desc = could not find container \"ad59b0471e3cce4ece2a3bc1e68f215f6d08b2bef3920668a883439a0c2a4b58\": container with ID starting with ad59b0471e3cce4ece2a3bc1e68f215f6d08b2bef3920668a883439a0c2a4b58 not found: ID does not exist"
Oct 30 01:25:41.508371 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.508328    6099 scope.go:122] "RemoveContainer" containerID="93d5c66001633e84d2952abc81e3193a7f97d5f060bee02e676d8b62ecc0481c"
Oct 30 01:25:41.520035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.520012    6099 scope.go:122] "RemoveContainer" containerID="93d5c66001633e84d2952abc81e3193a7f97d5f060bee02e676d8b62ecc0481c"
Oct 30 01:25:41.520440 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:25:41.520420    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d5c66001633e84d2952abc81e3193a7f97d5f060bee02e676d8b62ecc0481c\": container with ID starting with 93d5c66001633e84d2952abc81e3193a7f97d5f060bee02e676d8b62ecc0481c not found: ID does not exist" containerID="93d5c66001633e84d2952abc81e3193a7f97d5f060bee02e676d8b62ecc0481c"
Oct 30 01:25:41.520535 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.520449    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d5c66001633e84d2952abc81e3193a7f97d5f060bee02e676d8b62ecc0481c"} err="failed to get container status \"93d5c66001633e84d2952abc81e3193a7f97d5f060bee02e676d8b62ecc0481c\": rpc error: code = NotFound desc = could not find container \"93d5c66001633e84d2952abc81e3193a7f97d5f060bee02e676d8b62ecc0481c\": container with ID starting with 93d5c66001633e84d2952abc81e3193a7f97d5f060bee02e676d8b62ecc0481c not found: ID does not exist"
Oct 30 01:25:41.588255 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.588225    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.588255 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.588250    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.588255 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.588260    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0b7e93-d524-4443-98ca-a0993b9232c0-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.588473 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.588269    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.588473 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.588282    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.588473 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.588292    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227245db-8ac3-47c1-ae8c-ba5ff96202c6-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.588473 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.588299    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26062a6-cf7a-4faf-a955-5e408bc47c08-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.588473 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.588307    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bh8n\" (UniqueName: \"kubernetes.io/projected/f26062a6-cf7a-4faf-a955-5e408bc47c08-kube-api-access-9bh8n\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:41.788430 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.788397    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"]
Oct 30 01:25:41.792814 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.792787    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-zjvdb"]
Oct 30 01:25:41.826193 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.826166    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"]
Oct 30 01:25:41.832030 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.832000    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-l9q62"]
Oct 30 01:25:41.864556 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.864532    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"]
Oct 30 01:25:41.871636 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:41.871612    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-16c1bb-openshift-default-7546866c77-w2dwq"]
Oct 30 01:25:42.260527 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:42.260494    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="227245db-8ac3-47c1-ae8c-ba5ff96202c6" path="/var/lib/kubelet/pods/227245db-8ac3-47c1-ae8c-ba5ff96202c6/volumes"
Oct 30 01:25:42.261014 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:42.260999    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e0b7e93-d524-4443-98ca-a0993b9232c0" path="/var/lib/kubelet/pods/3e0b7e93-d524-4443-98ca-a0993b9232c0/volumes"
Oct 30 01:25:42.261457 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:42.261443    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f26062a6-cf7a-4faf-a955-5e408bc47c08" path="/var/lib/kubelet/pods/f26062a6-cf7a-4faf-a955-5e408bc47c08/volumes"
Oct 30 01:25:42.598997 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:42.598961    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"]
Oct 30 01:25:42.599512 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:42.599151    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj" podUID="8c2a8c69-5075-4f11-9920-2aeb2ee26ba8" containerName="discovery" containerID="cri-o://97c5ff2a1ca4c81eb8207a3636f412e526cbe2ae0fe0397c51c54be073c45c09" gracePeriod=30
Oct 30 01:25:42.863949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:42.863893    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:25:43.003145 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.003112    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-token\" (UniqueName: \"kubernetes.io/projected/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-token\") pod \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") "
Oct 30 01:25:43.003319 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.003157    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-kubeconfig\") pod \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") "
Oct 30 01:25:43.003394 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.003349    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-kube-api-access-srsf6\" (UniqueName: \"kubernetes.io/projected/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-kube-api-access-srsf6\") pod \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") "
Oct 30 01:25:43.003460 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.003427    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-cacerts\" (UniqueName: \"kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-cacerts\") pod \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") "
Oct 30 01:25:43.003524 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.003498    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-csr-dns-cert\") pod \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") "
Oct 30 01:25:43.003631 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.003606    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-local-certs\") pod \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") "
Oct 30 01:25:43.003756 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.003699    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-csr-ca-configmap\") pod \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\" (UID: \"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8\") "
Oct 30 01:25:43.004366 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.004310    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-csr-ca-configmap" pod "8c2a8c69-5075-4f11-9920-2aeb2ee26ba8" (UID: "8c2a8c69-5075-4f11-9920-2aeb2ee26ba8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:43.005593 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.005562    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-cacerts" pod "8c2a8c69-5075-4f11-9920-2aeb2ee26ba8" (UID: "8c2a8c69-5075-4f11-9920-2aeb2ee26ba8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:43.005730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.005691    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-kubeconfig" pod "8c2a8c69-5075-4f11-9920-2aeb2ee26ba8" (UID: "8c2a8c69-5075-4f11-9920-2aeb2ee26ba8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:43.005888 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.005854    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-token" pod "8c2a8c69-5075-4f11-9920-2aeb2ee26ba8" (UID: "8c2a8c69-5075-4f11-9920-2aeb2ee26ba8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:43.005967 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.005895    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-kube-api-access-srsf6" pod "8c2a8c69-5075-4f11-9920-2aeb2ee26ba8" (UID: "8c2a8c69-5075-4f11-9920-2aeb2ee26ba8"). InnerVolumeSpecName "kube-api-access-srsf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:43.005967 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.005898    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-csr-dns-cert" pod "8c2a8c69-5075-4f11-9920-2aeb2ee26ba8" (UID: "8c2a8c69-5075-4f11-9920-2aeb2ee26ba8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:43.006108 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.006087    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-local-certs" pod "8c2a8c69-5075-4f11-9920-2aeb2ee26ba8" (UID: "8c2a8c69-5075-4f11-9920-2aeb2ee26ba8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:43.105780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.105750    6099 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-local-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:43.105780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.105777    6099 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-csr-ca-configmap\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:43.105986 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.105789    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:43.105986 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.105799    6099 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-kubeconfig\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:43.105986 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.105806    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srsf6\" (UniqueName: \"kubernetes.io/projected/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-kube-api-access-srsf6\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:43.105986 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.105816    6099 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-cacerts\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:43.105986 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.105824    6099 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8-istio-csr-dns-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:43.460634 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.460598    6099 generic.go:356] "Generic (PLEG): container finished" podID="8c2a8c69-5075-4f11-9920-2aeb2ee26ba8" containerID="97c5ff2a1ca4c81eb8207a3636f412e526cbe2ae0fe0397c51c54be073c45c09" exitCode=0
Oct 30 01:25:43.460835 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.460652    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"
Oct 30 01:25:43.460835 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.460689    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj" event={"ID":"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8","Type":"ContainerDied","Data":"97c5ff2a1ca4c81eb8207a3636f412e526cbe2ae0fe0397c51c54be073c45c09"}
Oct 30 01:25:43.460835 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.460753    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj" event={"ID":"8c2a8c69-5075-4f11-9920-2aeb2ee26ba8","Type":"ContainerDied","Data":"0b50483b2480b76cc9fa62d3a6bbd8bc6a8233a7716ddd4db83c5f81e10e13c5"}
Oct 30 01:25:43.460835 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.460778    6099 scope.go:122] "RemoveContainer" containerID="97c5ff2a1ca4c81eb8207a3636f412e526cbe2ae0fe0397c51c54be073c45c09"
Oct 30 01:25:43.471854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.471835    6099 scope.go:122] "RemoveContainer" containerID="97c5ff2a1ca4c81eb8207a3636f412e526cbe2ae0fe0397c51c54be073c45c09"
Oct 30 01:25:43.472107 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:25:43.472089    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c5ff2a1ca4c81eb8207a3636f412e526cbe2ae0fe0397c51c54be073c45c09\": container with ID starting with 97c5ff2a1ca4c81eb8207a3636f412e526cbe2ae0fe0397c51c54be073c45c09 not found: ID does not exist" containerID="97c5ff2a1ca4c81eb8207a3636f412e526cbe2ae0fe0397c51c54be073c45c09"
Oct 30 01:25:43.472159 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.472115    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c5ff2a1ca4c81eb8207a3636f412e526cbe2ae0fe0397c51c54be073c45c09"} err="failed to get container status \"97c5ff2a1ca4c81eb8207a3636f412e526cbe2ae0fe0397c51c54be073c45c09\": rpc error: code = NotFound desc = could not find container \"97c5ff2a1ca4c81eb8207a3636f412e526cbe2ae0fe0397c51c54be073c45c09\": container with ID starting with 97c5ff2a1ca4c81eb8207a3636f412e526cbe2ae0fe0397c51c54be073c45c09 not found: ID does not exist"
Oct 30 01:25:43.525881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.525843    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"]
Oct 30 01:25:43.537771 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:43.537745    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvbtj"]
Oct 30 01:25:44.259925 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:44.259894    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c2a8c69-5075-4f11-9920-2aeb2ee26ba8" path="/var/lib/kubelet/pods/8c2a8c69-5075-4f11-9920-2aeb2ee26ba8/volumes"
Oct 30 01:25:46.629370 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:25:46.629328    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:25:47.925797 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:47.925768    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42"]
Oct 30 01:25:49.496724 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:49.496583    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"]
Oct 30 01:25:49.497550 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:49.497516    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz" podUID="330745dd-20f6-4e76-8513-3abf69325b80" containerName="maas-api" containerID="cri-o://2cb8b78c1f0879cbd38771a98e3160409f70d38abcdb61f0c8ee8a890aeb2434" gracePeriod=30
Oct 30 01:25:49.801524 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:49.801493    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:25:49.897352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:49.897313    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330745dd-20f6-4e76-8513-3abf69325b80-kube-api-access-9ddns\" (UniqueName: \"kubernetes.io/projected/330745dd-20f6-4e76-8513-3abf69325b80-kube-api-access-9ddns\") pod \"330745dd-20f6-4e76-8513-3abf69325b80\" (UID: \"330745dd-20f6-4e76-8513-3abf69325b80\") "
Oct 30 01:25:49.897555 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:49.897376    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/330745dd-20f6-4e76-8513-3abf69325b80-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/330745dd-20f6-4e76-8513-3abf69325b80-maas-api-tls\") pod \"330745dd-20f6-4e76-8513-3abf69325b80\" (UID: \"330745dd-20f6-4e76-8513-3abf69325b80\") "
Oct 30 01:25:49.897555 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:49.897473    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/330745dd-20f6-4e76-8513-3abf69325b80-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/330745dd-20f6-4e76-8513-3abf69325b80-openshift-service-ca\") pod \"330745dd-20f6-4e76-8513-3abf69325b80\" (UID: \"330745dd-20f6-4e76-8513-3abf69325b80\") "
Oct 30 01:25:49.897555 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:49.897520    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/330745dd-20f6-4e76-8513-3abf69325b80-metrics-certs\" (UniqueName: \"kubernetes.io/secret/330745dd-20f6-4e76-8513-3abf69325b80-metrics-certs\") pod \"330745dd-20f6-4e76-8513-3abf69325b80\" (UID: \"330745dd-20f6-4e76-8513-3abf69325b80\") "
Oct 30 01:25:49.897947 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:49.897812    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330745dd-20f6-4e76-8513-3abf69325b80-openshift-service-ca" pod "330745dd-20f6-4e76-8513-3abf69325b80" (UID: "330745dd-20f6-4e76-8513-3abf69325b80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:49.898117 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:49.897967    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/330745dd-20f6-4e76-8513-3abf69325b80-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:49.900090 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:49.900064    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330745dd-20f6-4e76-8513-3abf69325b80-kube-api-access-9ddns" pod "330745dd-20f6-4e76-8513-3abf69325b80" (UID: "330745dd-20f6-4e76-8513-3abf69325b80"). InnerVolumeSpecName "kube-api-access-9ddns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:49.902183 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:49.902151    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/330745dd-20f6-4e76-8513-3abf69325b80-metrics-certs" pod "330745dd-20f6-4e76-8513-3abf69325b80" (UID: "330745dd-20f6-4e76-8513-3abf69325b80"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:49.902315 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:49.902253    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/330745dd-20f6-4e76-8513-3abf69325b80-maas-api-tls" pod "330745dd-20f6-4e76-8513-3abf69325b80" (UID: "330745dd-20f6-4e76-8513-3abf69325b80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:49.998961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:49.998925    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/330745dd-20f6-4e76-8513-3abf69325b80-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:49.998961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:49.998958    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ddns\" (UniqueName: \"kubernetes.io/projected/330745dd-20f6-4e76-8513-3abf69325b80-kube-api-access-9ddns\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:49.998961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:49.998969    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/330745dd-20f6-4e76-8513-3abf69325b80-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:50.531791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:50.531755    6099 generic.go:356] "Generic (PLEG): container finished" podID="330745dd-20f6-4e76-8513-3abf69325b80" containerID="2cb8b78c1f0879cbd38771a98e3160409f70d38abcdb61f0c8ee8a890aeb2434" exitCode=0
Oct 30 01:25:50.532286 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:50.531799    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"
Oct 30 01:25:50.532286 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:50.531836    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz" event={"ID":"330745dd-20f6-4e76-8513-3abf69325b80","Type":"ContainerDied","Data":"2cb8b78c1f0879cbd38771a98e3160409f70d38abcdb61f0c8ee8a890aeb2434"}
Oct 30 01:25:50.532286 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:50.531892    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz" event={"ID":"330745dd-20f6-4e76-8513-3abf69325b80","Type":"ContainerDied","Data":"e9ee3731a13291c3aafabaeb926cb02f74a939d67672995b9e354010946e6f76"}
Oct 30 01:25:50.532286 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:50.531920    6099 scope.go:122] "RemoveContainer" containerID="2cb8b78c1f0879cbd38771a98e3160409f70d38abcdb61f0c8ee8a890aeb2434"
Oct 30 01:25:50.542330 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:50.542291    6099 scope.go:122] "RemoveContainer" containerID="2cb8b78c1f0879cbd38771a98e3160409f70d38abcdb61f0c8ee8a890aeb2434"
Oct 30 01:25:50.542567 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:25:50.542546    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb8b78c1f0879cbd38771a98e3160409f70d38abcdb61f0c8ee8a890aeb2434\": container with ID starting with 2cb8b78c1f0879cbd38771a98e3160409f70d38abcdb61f0c8ee8a890aeb2434 not found: ID does not exist" containerID="2cb8b78c1f0879cbd38771a98e3160409f70d38abcdb61f0c8ee8a890aeb2434"
Oct 30 01:25:50.542638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:50.542578    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb8b78c1f0879cbd38771a98e3160409f70d38abcdb61f0c8ee8a890aeb2434"} err="failed to get container status \"2cb8b78c1f0879cbd38771a98e3160409f70d38abcdb61f0c8ee8a890aeb2434\": rpc error: code = NotFound desc = could not find container \"2cb8b78c1f0879cbd38771a98e3160409f70d38abcdb61f0c8ee8a890aeb2434\": container with ID starting with 2cb8b78c1f0879cbd38771a98e3160409f70d38abcdb61f0c8ee8a890aeb2434 not found: ID does not exist"
Oct 30 01:25:50.584253 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:50.584013    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"]
Oct 30 01:25:50.595114 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:50.595086    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e24a5-8697996cf4-vmpfz"]
Oct 30 01:25:51.624169 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624136    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz"]
Oct 30 01:25:51.624652 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624637    6099 cpu_manager.go:414] "containerMap: removing container" podUID="227245db-8ac3-47c1-ae8c-ba5ff96202c6" containerName="istio-proxy"
Oct 30 01:25:51.624699 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624653    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="227245db-8ac3-47c1-ae8c-ba5ff96202c6" containerName="istio-proxy"
Oct 30 01:25:51.624699 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624666    6099 cpu_manager.go:414] "containerMap: removing container" podUID="330745dd-20f6-4e76-8513-3abf69325b80" containerName="maas-api"
Oct 30 01:25:51.624699 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624672    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330745dd-20f6-4e76-8513-3abf69325b80" containerName="maas-api"
Oct 30 01:25:51.624699 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624689    6099 cpu_manager.go:414] "containerMap: removing container" podUID="8c2a8c69-5075-4f11-9920-2aeb2ee26ba8" containerName="discovery"
Oct 30 01:25:51.624699 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624695    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c2a8c69-5075-4f11-9920-2aeb2ee26ba8" containerName="discovery"
Oct 30 01:25:51.624887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624723    6099 cpu_manager.go:414] "containerMap: removing container" podUID="f26062a6-cf7a-4faf-a955-5e408bc47c08" containerName="istio-proxy"
Oct 30 01:25:51.624887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624728    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f26062a6-cf7a-4faf-a955-5e408bc47c08" containerName="istio-proxy"
Oct 30 01:25:51.624887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624733    6099 cpu_manager.go:414] "containerMap: removing container" podUID="79665e1a-c51c-4623-8c73-f505c73bf38b" containerName="revoke-keys"
Oct 30 01:25:51.624887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624738    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79665e1a-c51c-4623-8c73-f505c73bf38b" containerName="revoke-keys"
Oct 30 01:25:51.624887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624750    6099 cpu_manager.go:414] "containerMap: removing container" podUID="3e0b7e93-d524-4443-98ca-a0993b9232c0" containerName="istio-proxy"
Oct 30 01:25:51.624887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624757    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e0b7e93-d524-4443-98ca-a0993b9232c0" containerName="istio-proxy"
Oct 30 01:25:51.624887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624829    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="227245db-8ac3-47c1-ae8c-ba5ff96202c6" containerName="istio-proxy"
Oct 30 01:25:51.624887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624843    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="330745dd-20f6-4e76-8513-3abf69325b80" containerName="maas-api"
Oct 30 01:25:51.624887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624851    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c2a8c69-5075-4f11-9920-2aeb2ee26ba8" containerName="discovery"
Oct 30 01:25:51.624887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624860    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f26062a6-cf7a-4faf-a955-5e408bc47c08" containerName="istio-proxy"
Oct 30 01:25:51.624887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624866    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="79665e1a-c51c-4623-8c73-f505c73bf38b" containerName="revoke-keys"
Oct 30 01:25:51.624887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.624872    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e0b7e93-d524-4443-98ca-a0993b9232c0" containerName="istio-proxy"
Oct 30 01:25:51.629975 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.629951    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz"
Oct 30 01:25:51.649189 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.649164    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g"]
Oct 30 01:25:51.655854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.655806    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:25:51.655854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.655833    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:25:51.655854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.655835    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:25:51.658282 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.658259    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz"]
Oct 30 01:25:51.658421 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.658377    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g"
Oct 30 01:25:51.659273 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.659252    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:25:51.698976 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.698943    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g"]
Oct 30 01:25:51.820668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.820623    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac579e0b-c335-44da-82d1-760d7066ccc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4609f84-h89sz\" (UID: \"ac579e0b-c335-44da-82d1-760d7066ccc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz"
Oct 30 01:25:51.820668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.820677    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nqgh\" (UniqueName: \"kubernetes.io/projected/ac579e0b-c335-44da-82d1-760d7066ccc7-kube-api-access-8nqgh\") pod \"maas-api-revoke-sub-e4609f84-h89sz\" (UID: \"ac579e0b-c335-44da-82d1-760d7066ccc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz"
Oct 30 01:25:51.820950 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.820779    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e515a259-784c-4778-af11-4a0088e3cef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3f9826e-khr2g\" (UID: \"e515a259-784c-4778-af11-4a0088e3cef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g"
Oct 30 01:25:51.820950 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.820843    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac579e0b-c335-44da-82d1-760d7066ccc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4609f84-h89sz\" (UID: \"ac579e0b-c335-44da-82d1-760d7066ccc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz"
Oct 30 01:25:51.820950 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.820937    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngnjc\" (UniqueName: \"kubernetes.io/projected/e515a259-784c-4778-af11-4a0088e3cef8-kube-api-access-ngnjc\") pod \"maas-api-revoke-sub-d3f9826e-khr2g\" (UID: \"e515a259-784c-4778-af11-4a0088e3cef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g"
Oct 30 01:25:51.821139 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.820998    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e515a259-784c-4778-af11-4a0088e3cef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3f9826e-khr2g\" (UID: \"e515a259-784c-4778-af11-4a0088e3cef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g"
Oct 30 01:25:51.921670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.921578    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nqgh\" (UniqueName: \"kubernetes.io/projected/ac579e0b-c335-44da-82d1-760d7066ccc7-kube-api-access-8nqgh\") pod \"maas-api-revoke-sub-e4609f84-h89sz\" (UID: \"ac579e0b-c335-44da-82d1-760d7066ccc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz"
Oct 30 01:25:51.921670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.921619    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e515a259-784c-4778-af11-4a0088e3cef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3f9826e-khr2g\" (UID: \"e515a259-784c-4778-af11-4a0088e3cef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g"
Oct 30 01:25:51.921670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.921665    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac579e0b-c335-44da-82d1-760d7066ccc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4609f84-h89sz\" (UID: \"ac579e0b-c335-44da-82d1-760d7066ccc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz"
Oct 30 01:25:51.922077 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.921819    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnjc\" (UniqueName: \"kubernetes.io/projected/e515a259-784c-4778-af11-4a0088e3cef8-kube-api-access-ngnjc\") pod \"maas-api-revoke-sub-d3f9826e-khr2g\" (UID: \"e515a259-784c-4778-af11-4a0088e3cef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g"
Oct 30 01:25:51.922077 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.921877    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e515a259-784c-4778-af11-4a0088e3cef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3f9826e-khr2g\" (UID: \"e515a259-784c-4778-af11-4a0088e3cef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g"
Oct 30 01:25:51.922077 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.922053    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac579e0b-c335-44da-82d1-760d7066ccc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4609f84-h89sz\" (UID: \"ac579e0b-c335-44da-82d1-760d7066ccc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz"
Oct 30 01:25:51.922858 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.922831    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e515a259-784c-4778-af11-4a0088e3cef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3f9826e-khr2g\" (UID: \"e515a259-784c-4778-af11-4a0088e3cef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g"
Oct 30 01:25:51.922981 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.922864    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac579e0b-c335-44da-82d1-760d7066ccc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4609f84-h89sz\" (UID: \"ac579e0b-c335-44da-82d1-760d7066ccc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz"
Oct 30 01:25:51.924505 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.924480    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e515a259-784c-4778-af11-4a0088e3cef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3f9826e-khr2g\" (UID: \"e515a259-784c-4778-af11-4a0088e3cef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g"
Oct 30 01:25:51.924505 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.924498    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac579e0b-c335-44da-82d1-760d7066ccc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4609f84-h89sz\" (UID: \"ac579e0b-c335-44da-82d1-760d7066ccc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz"
Oct 30 01:25:51.944649 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.944625    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngnjc\" (UniqueName: \"kubernetes.io/projected/e515a259-784c-4778-af11-4a0088e3cef8-kube-api-access-ngnjc\") pod \"maas-api-revoke-sub-d3f9826e-khr2g\" (UID: \"e515a259-784c-4778-af11-4a0088e3cef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g"
Oct 30 01:25:51.944649 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.944642    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nqgh\" (UniqueName: \"kubernetes.io/projected/ac579e0b-c335-44da-82d1-760d7066ccc7-kube-api-access-8nqgh\") pod \"maas-api-revoke-sub-e4609f84-h89sz\" (UID: \"ac579e0b-c335-44da-82d1-760d7066ccc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz"
Oct 30 01:25:51.953442 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:51.953423    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz"
Oct 30 01:25:52.002589 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:52.002561    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g"
Oct 30 01:25:52.260689 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:52.260617    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="330745dd-20f6-4e76-8513-3abf69325b80" path="/var/lib/kubelet/pods/330745dd-20f6-4e76-8513-3abf69325b80/volumes"
Oct 30 01:25:52.292429 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:52.292405    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz"]
Oct 30 01:25:52.295244 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:25:52.295214    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac579e0b_c335_44da_82d1_760d7066ccc7.slice/crio-3b7e331100f8b61245b2c205fdd9320439710b2982d40f04f3806bd86dcf1a3b WatchSource:0}: Error finding container 3b7e331100f8b61245b2c205fdd9320439710b2982d40f04f3806bd86dcf1a3b: Status 404 returned error can't find the container with id 3b7e331100f8b61245b2c205fdd9320439710b2982d40f04f3806bd86dcf1a3b
Oct 30 01:25:52.351906 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:52.351554    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g"]
Oct 30 01:25:52.352572 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:25:52.352553    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode515a259_784c_4778_af11_4a0088e3cef8.slice/crio-b196c05d91ebded24958feb44519ef015537a6591b22abc98a8b63fdbdfe6f68 WatchSource:0}: Error finding container b196c05d91ebded24958feb44519ef015537a6591b22abc98a8b63fdbdfe6f68: Status 404 returned error can't find the container with id b196c05d91ebded24958feb44519ef015537a6591b22abc98a8b63fdbdfe6f68
Oct 30 01:25:52.545889 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:52.545853    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g" event={"ID":"e515a259-784c-4778-af11-4a0088e3cef8","Type":"ContainerStarted","Data":"b196c05d91ebded24958feb44519ef015537a6591b22abc98a8b63fdbdfe6f68"}
Oct 30 01:25:52.547254 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:52.547224    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz" event={"ID":"ac579e0b-c335-44da-82d1-760d7066ccc7","Type":"ContainerStarted","Data":"3b7e331100f8b61245b2c205fdd9320439710b2982d40f04f3806bd86dcf1a3b"}
Oct 30 01:25:52.927039 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:52.926946    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42" podUID="53ccfecd-0ce5-4382-a5e6-e8acd989e4e1" containerName="payload-pre-processing" containerID="cri-o://f5720116286575ea8d5b331f78ac5238b62d9dd5996bddacfd7567413292bdb5" gracePeriod=25
Oct 30 01:25:53.042947 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:25:53.042919    6099 logging.go:55] [core] [Channel #8443 SubChannel #8444]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.13:50052", ServerName: "10.134.1.13:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.13:50052: connect: connection refused"
Oct 30 01:25:53.225157 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.225117    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42"
Oct 30 01:25:53.234881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.234857    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53ccfecd-0ce5-4382-a5e6-e8acd989e4e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ccfecd-0ce5-4382-a5e6-e8acd989e4e1-plugins-config-volume\") pod \"53ccfecd-0ce5-4382-a5e6-e8acd989e4e1\" (UID: \"53ccfecd-0ce5-4382-a5e6-e8acd989e4e1\") "
Oct 30 01:25:53.235012 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.234919    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53ccfecd-0ce5-4382-a5e6-e8acd989e4e1-kube-api-access-k7dk4\" (UniqueName: \"kubernetes.io/projected/53ccfecd-0ce5-4382-a5e6-e8acd989e4e1-kube-api-access-k7dk4\") pod \"53ccfecd-0ce5-4382-a5e6-e8acd989e4e1\" (UID: \"53ccfecd-0ce5-4382-a5e6-e8acd989e4e1\") "
Oct 30 01:25:53.235209 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.235187    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ccfecd-0ce5-4382-a5e6-e8acd989e4e1-plugins-config-volume" pod "53ccfecd-0ce5-4382-a5e6-e8acd989e4e1" (UID: "53ccfecd-0ce5-4382-a5e6-e8acd989e4e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:53.237008 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.236973    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ccfecd-0ce5-4382-a5e6-e8acd989e4e1-kube-api-access-k7dk4" pod "53ccfecd-0ce5-4382-a5e6-e8acd989e4e1" (UID: "53ccfecd-0ce5-4382-a5e6-e8acd989e4e1"). InnerVolumeSpecName "kube-api-access-k7dk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:53.336205 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.336177    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ccfecd-0ce5-4382-a5e6-e8acd989e4e1-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:53.336205 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.336204    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7dk4\" (UniqueName: \"kubernetes.io/projected/53ccfecd-0ce5-4382-a5e6-e8acd989e4e1-kube-api-access-k7dk4\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:53.554865 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.554830    6099 generic.go:356] "Generic (PLEG): container finished" podID="53ccfecd-0ce5-4382-a5e6-e8acd989e4e1" containerID="f5720116286575ea8d5b331f78ac5238b62d9dd5996bddacfd7567413292bdb5" exitCode=0
Oct 30 01:25:53.555035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.554881    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42"
Oct 30 01:25:53.555035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.554913    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42" event={"ID":"53ccfecd-0ce5-4382-a5e6-e8acd989e4e1","Type":"ContainerDied","Data":"f5720116286575ea8d5b331f78ac5238b62d9dd5996bddacfd7567413292bdb5"}
Oct 30 01:25:53.555035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.554977    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42" event={"ID":"53ccfecd-0ce5-4382-a5e6-e8acd989e4e1","Type":"ContainerDied","Data":"1aad9c3371e8c4c67d01a1e4c35701477ef5d0f79729f70f2622a6a4481a5bea"}
Oct 30 01:25:53.555035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.555008    6099 scope.go:122] "RemoveContainer" containerID="f5720116286575ea8d5b331f78ac5238b62d9dd5996bddacfd7567413292bdb5"
Oct 30 01:25:53.556605 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.556576    6099 generic.go:356] "Generic (PLEG): container finished" podID="ac579e0b-c335-44da-82d1-760d7066ccc7" containerID="9c3fd62156290e5dc06bedae110b000dd70d50521a166a8bb7906511a93392dc" exitCode=0
Oct 30 01:25:53.556738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.556658    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz" event={"ID":"ac579e0b-c335-44da-82d1-760d7066ccc7","Type":"ContainerDied","Data":"9c3fd62156290e5dc06bedae110b000dd70d50521a166a8bb7906511a93392dc"}
Oct 30 01:25:53.558515 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.558492    6099 generic.go:356] "Generic (PLEG): container finished" podID="e515a259-784c-4778-af11-4a0088e3cef8" containerID="3ac0e43f7867db298c153a83538b361d4aea218f9abacf3416db762bd1c1bd14" exitCode=0
Oct 30 01:25:53.558639 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.558555    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g" event={"ID":"e515a259-784c-4778-af11-4a0088e3cef8","Type":"ContainerDied","Data":"3ac0e43f7867db298c153a83538b361d4aea218f9abacf3416db762bd1c1bd14"}
Oct 30 01:25:53.591975 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.591950    6099 scope.go:122] "RemoveContainer" containerID="f5720116286575ea8d5b331f78ac5238b62d9dd5996bddacfd7567413292bdb5"
Oct 30 01:25:53.592304 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:25:53.592284    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5720116286575ea8d5b331f78ac5238b62d9dd5996bddacfd7567413292bdb5\": container with ID starting with f5720116286575ea8d5b331f78ac5238b62d9dd5996bddacfd7567413292bdb5 not found: ID does not exist" containerID="f5720116286575ea8d5b331f78ac5238b62d9dd5996bddacfd7567413292bdb5"
Oct 30 01:25:53.592370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.592313    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5720116286575ea8d5b331f78ac5238b62d9dd5996bddacfd7567413292bdb5"} err="failed to get container status \"f5720116286575ea8d5b331f78ac5238b62d9dd5996bddacfd7567413292bdb5\": rpc error: code = NotFound desc = could not find container \"f5720116286575ea8d5b331f78ac5238b62d9dd5996bddacfd7567413292bdb5\": container with ID starting with f5720116286575ea8d5b331f78ac5238b62d9dd5996bddacfd7567413292bdb5 not found: ID does not exist"
Oct 30 01:25:53.725697 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.725621    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42"]
Oct 30 01:25:53.729363 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.729339    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42"]
Oct 30 01:25:53.790605 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:53.771957    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:25:54.042738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.042679    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e24a5-56d4d96b5f-4lq42" podUID="53ccfecd-0ce5-4382-a5e6-e8acd989e4e1" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.13:50052\" within 1s: context deadline exceeded"
Oct 30 01:25:54.260377 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.260345    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53ccfecd-0ce5-4382-a5e6-e8acd989e4e1" path="/var/lib/kubelet/pods/53ccfecd-0ce5-4382-a5e6-e8acd989e4e1/volumes"
Oct 30 01:25:54.724574 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.724545    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz"
Oct 30 01:25:54.750543 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.750518    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac579e0b-c335-44da-82d1-760d7066ccc7-kube-api-access-8nqgh\" (UniqueName: \"kubernetes.io/projected/ac579e0b-c335-44da-82d1-760d7066ccc7-kube-api-access-8nqgh\") pod \"ac579e0b-c335-44da-82d1-760d7066ccc7\" (UID: \"ac579e0b-c335-44da-82d1-760d7066ccc7\") "
Oct 30 01:25:54.750685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.750599    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac579e0b-c335-44da-82d1-760d7066ccc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac579e0b-c335-44da-82d1-760d7066ccc7-maas-api-xks-ca\") pod \"ac579e0b-c335-44da-82d1-760d7066ccc7\" (UID: \"ac579e0b-c335-44da-82d1-760d7066ccc7\") "
Oct 30 01:25:54.750792 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.750730    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac579e0b-c335-44da-82d1-760d7066ccc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac579e0b-c335-44da-82d1-760d7066ccc7-maas-api-openshift-ca\") pod \"ac579e0b-c335-44da-82d1-760d7066ccc7\" (UID: \"ac579e0b-c335-44da-82d1-760d7066ccc7\") "
Oct 30 01:25:54.751200 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.751173    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac579e0b-c335-44da-82d1-760d7066ccc7-maas-api-openshift-ca" pod "ac579e0b-c335-44da-82d1-760d7066ccc7" (UID: "ac579e0b-c335-44da-82d1-760d7066ccc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:54.751534 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.751519    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g"
Oct 30 01:25:54.752740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.752690    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac579e0b-c335-44da-82d1-760d7066ccc7-maas-api-xks-ca" pod "ac579e0b-c335-44da-82d1-760d7066ccc7" (UID: "ac579e0b-c335-44da-82d1-760d7066ccc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:54.753117 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.753099    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac579e0b-c335-44da-82d1-760d7066ccc7-kube-api-access-8nqgh" pod "ac579e0b-c335-44da-82d1-760d7066ccc7" (UID: "ac579e0b-c335-44da-82d1-760d7066ccc7"). InnerVolumeSpecName "kube-api-access-8nqgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:54.852015 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.851922    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e515a259-784c-4778-af11-4a0088e3cef8-kube-api-access-ngnjc\" (UniqueName: \"kubernetes.io/projected/e515a259-784c-4778-af11-4a0088e3cef8-kube-api-access-ngnjc\") pod \"e515a259-784c-4778-af11-4a0088e3cef8\" (UID: \"e515a259-784c-4778-af11-4a0088e3cef8\") "
Oct 30 01:25:54.852211 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.852016    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e515a259-784c-4778-af11-4a0088e3cef8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e515a259-784c-4778-af11-4a0088e3cef8-maas-api-xks-ca\") pod \"e515a259-784c-4778-af11-4a0088e3cef8\" (UID: \"e515a259-784c-4778-af11-4a0088e3cef8\") "
Oct 30 01:25:54.852211 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.852078    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e515a259-784c-4778-af11-4a0088e3cef8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e515a259-784c-4778-af11-4a0088e3cef8-maas-api-openshift-ca\") pod \"e515a259-784c-4778-af11-4a0088e3cef8\" (UID: \"e515a259-784c-4778-af11-4a0088e3cef8\") "
Oct 30 01:25:54.852534 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.852512    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nqgh\" (UniqueName: \"kubernetes.io/projected/ac579e0b-c335-44da-82d1-760d7066ccc7-kube-api-access-8nqgh\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:54.852616 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.852540    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac579e0b-c335-44da-82d1-760d7066ccc7-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:54.852616 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.852556    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac579e0b-c335-44da-82d1-760d7066ccc7-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:54.852616 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.852561    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e515a259-784c-4778-af11-4a0088e3cef8-maas-api-openshift-ca" pod "e515a259-784c-4778-af11-4a0088e3cef8" (UID: "e515a259-784c-4778-af11-4a0088e3cef8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:54.854454 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.854424    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e515a259-784c-4778-af11-4a0088e3cef8-maas-api-xks-ca" pod "e515a259-784c-4778-af11-4a0088e3cef8" (UID: "e515a259-784c-4778-af11-4a0088e3cef8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:54.854454 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.854425    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e515a259-784c-4778-af11-4a0088e3cef8-kube-api-access-ngnjc" pod "e515a259-784c-4778-af11-4a0088e3cef8" (UID: "e515a259-784c-4778-af11-4a0088e3cef8"). InnerVolumeSpecName "kube-api-access-ngnjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:54.953995 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.953955    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngnjc\" (UniqueName: \"kubernetes.io/projected/e515a259-784c-4778-af11-4a0088e3cef8-kube-api-access-ngnjc\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:54.953995 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.953994    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e515a259-784c-4778-af11-4a0088e3cef8-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:54.954202 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:54.954010    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e515a259-784c-4778-af11-4a0088e3cef8-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:55.576643 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:55.576613    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz"
Oct 30 01:25:55.577169 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:55.576648    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz" event={"ID":"ac579e0b-c335-44da-82d1-760d7066ccc7","Type":"ContainerDied","Data":"3b7e331100f8b61245b2c205fdd9320439710b2982d40f04f3806bd86dcf1a3b"}
Oct 30 01:25:55.577169 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:55.576696    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b7e331100f8b61245b2c205fdd9320439710b2982d40f04f3806bd86dcf1a3b"
Oct 30 01:25:55.578377 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:55.578355    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g"
Oct 30 01:25:55.578502 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:55.578385    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g" event={"ID":"e515a259-784c-4778-af11-4a0088e3cef8","Type":"ContainerDied","Data":"b196c05d91ebded24958feb44519ef015537a6591b22abc98a8b63fdbdfe6f68"}
Oct 30 01:25:55.578502 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:55.578434    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b196c05d91ebded24958feb44519ef015537a6591b22abc98a8b63fdbdfe6f68"
Oct 30 01:25:57.265456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.265413    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"]
Oct 30 01:25:57.265997 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.265813    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2" podUID="2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca" containerName="istio-proxy" containerID="cri-o://4847907fbfb9326367fc14657c0215056ea37a52cbdc03d808cddfb31db65231" gracePeriod=30
Oct 30 01:25:57.592294 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.592266    6099 generic.go:356] "Generic (PLEG): container finished" podID="2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca" containerID="4847907fbfb9326367fc14657c0215056ea37a52cbdc03d808cddfb31db65231" exitCode=0
Oct 30 01:25:57.592473 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.592352    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2" event={"ID":"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca","Type":"ContainerDied","Data":"4847907fbfb9326367fc14657c0215056ea37a52cbdc03d808cddfb31db65231"}
Oct 30 01:25:57.719752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.719730    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:25:57.784030 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.783953    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-kube-api-access-m7pvh\" (UniqueName: \"kubernetes.io/projected/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-kube-api-access-m7pvh\") pod \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") "
Oct 30 01:25:57.784030 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.783997    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-envoy\") pod \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") "
Oct 30 01:25:57.784275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.784075    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-token\" (UniqueName: \"kubernetes.io/projected/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-token\") pod \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") "
Oct 30 01:25:57.784275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.784137    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-data\") pod \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") "
Oct 30 01:25:57.784275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.784181    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-credential-socket\") pod \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") "
Oct 30 01:25:57.784275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.784237    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istiod-ca-cert\") pod \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") "
Oct 30 01:25:57.784511 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.784292    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-workload-certs\") pod \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") "
Oct 30 01:25:57.784511 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.784379    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-podinfo\") pod \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") "
Oct 30 01:25:57.784511 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.784416    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-workload-socket\") pod \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\" (UID: \"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca\") "
Oct 30 01:25:57.785520 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.784533    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-credential-socket" pod "2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca" (UID: "2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:57.785520 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.784770    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-workload-socket" pod "2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca" (UID: "2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:57.785520 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.784961    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:57.785520 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.784989    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:57.785520 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.785336    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istiod-ca-cert" pod "2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca" (UID: "2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:57.785520 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.785329    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-data" pod "2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca" (UID: "2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:57.785520 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.785392    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-workload-certs" pod "2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca" (UID: "2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:57.787830 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.787804    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-envoy" pod "2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca" (UID: "2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:57.787830 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.787823    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-kube-api-access-m7pvh" pod "2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca" (UID: "2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca"). InnerVolumeSpecName "kube-api-access-m7pvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:57.787973 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.787822    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-podinfo" pod "2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca" (UID: "2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:25:57.787973 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.787841    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-token" pod "2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca" (UID: "2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:57.885644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.885612    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:57.885644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.885639    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:57.885644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.885647    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:57.885872 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.885656    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7pvh\" (UniqueName: \"kubernetes.io/projected/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-kube-api-access-m7pvh\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:57.885872 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.885667    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:57.885872 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.885675    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:57.885872 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.885682    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:25:57.902192 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.902165    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77"]
Oct 30 01:25:57.902687 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.902669    6099 cpu_manager.go:414] "containerMap: removing container" podUID="ac579e0b-c335-44da-82d1-760d7066ccc7" containerName="revoke-keys"
Oct 30 01:25:57.902766 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.902690    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac579e0b-c335-44da-82d1-760d7066ccc7" containerName="revoke-keys"
Oct 30 01:25:57.902766 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.902718    6099 cpu_manager.go:414] "containerMap: removing container" podUID="2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca" containerName="istio-proxy"
Oct 30 01:25:57.902766 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.902726    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca" containerName="istio-proxy"
Oct 30 01:25:57.902766 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.902753    6099 cpu_manager.go:414] "containerMap: removing container" podUID="e515a259-784c-4778-af11-4a0088e3cef8" containerName="revoke-keys"
Oct 30 01:25:57.902766 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.902760    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e515a259-784c-4778-af11-4a0088e3cef8" containerName="revoke-keys"
Oct 30 01:25:57.902927 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.902778    6099 cpu_manager.go:414] "containerMap: removing container" podUID="53ccfecd-0ce5-4382-a5e6-e8acd989e4e1" containerName="payload-pre-processing"
Oct 30 01:25:57.902927 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.902787    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ccfecd-0ce5-4382-a5e6-e8acd989e4e1" containerName="payload-pre-processing"
Oct 30 01:25:57.902927 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.902876    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac579e0b-c335-44da-82d1-760d7066ccc7" containerName="revoke-keys"
Oct 30 01:25:57.902927 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.902883    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca" containerName="istio-proxy"
Oct 30 01:25:57.902927 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.902893    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e515a259-784c-4778-af11-4a0088e3cef8" containerName="revoke-keys"
Oct 30 01:25:57.902927 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.902899    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ccfecd-0ce5-4382-a5e6-e8acd989e4e1" containerName="payload-pre-processing"
Oct 30 01:25:57.906244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.906230    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77"
Oct 30 01:25:57.928023 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.925983    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77"]
Oct 30 01:25:57.937265 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.937241    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:25:57.937265 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.937259    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:25:57.964322 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.964299    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:25:57.964485 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.964469    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:25:57.987065 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.987035    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdchp\" (UniqueName: \"kubernetes.io/projected/a1156317-e78b-4e13-a989-fe28b628cde1-kube-api-access-pdchp\") pod \"maas-api-revoke-keys-e2e-worker-w5-89a676-84x77\" (UID: \"a1156317-e78b-4e13-a989-fe28b628cde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77"
Oct 30 01:25:57.987205 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.987186    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1156317-e78b-4e13-a989-fe28b628cde1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-89a676-84x77\" (UID: \"a1156317-e78b-4e13-a989-fe28b628cde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77"
Oct 30 01:25:57.987284 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:57.987254    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1156317-e78b-4e13-a989-fe28b628cde1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-89a676-84x77\" (UID: \"a1156317-e78b-4e13-a989-fe28b628cde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77"
Oct 30 01:25:58.088820 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:58.088791    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1156317-e78b-4e13-a989-fe28b628cde1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-89a676-84x77\" (UID: \"a1156317-e78b-4e13-a989-fe28b628cde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77"
Oct 30 01:25:58.088959 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:58.088848    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdchp\" (UniqueName: \"kubernetes.io/projected/a1156317-e78b-4e13-a989-fe28b628cde1-kube-api-access-pdchp\") pod \"maas-api-revoke-keys-e2e-worker-w5-89a676-84x77\" (UID: \"a1156317-e78b-4e13-a989-fe28b628cde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77"
Oct 30 01:25:58.088959 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:58.088912    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1156317-e78b-4e13-a989-fe28b628cde1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-89a676-84x77\" (UID: \"a1156317-e78b-4e13-a989-fe28b628cde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77"
Oct 30 01:25:58.089404 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:58.089384    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1156317-e78b-4e13-a989-fe28b628cde1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-89a676-84x77\" (UID: \"a1156317-e78b-4e13-a989-fe28b628cde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77"
Oct 30 01:25:58.091260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:58.091236    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1156317-e78b-4e13-a989-fe28b628cde1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-89a676-84x77\" (UID: \"a1156317-e78b-4e13-a989-fe28b628cde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77"
Oct 30 01:25:58.111511 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:58.111491    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdchp\" (UniqueName: \"kubernetes.io/projected/a1156317-e78b-4e13-a989-fe28b628cde1-kube-api-access-pdchp\") pod \"maas-api-revoke-keys-e2e-worker-w5-89a676-84x77\" (UID: \"a1156317-e78b-4e13-a989-fe28b628cde1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77"
Oct 30 01:25:58.220077 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:58.220047    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77"
Oct 30 01:25:58.518950 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:58.518918    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77"]
Oct 30 01:25:58.519931 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:25:58.519905    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1156317_e78b_4e13_a989_fe28b628cde1.slice/crio-b3695c431bf15fd0e09e00efe76f0b5be8315502c6149b71776e9bd3ff541880 WatchSource:0}: Error finding container b3695c431bf15fd0e09e00efe76f0b5be8315502c6149b71776e9bd3ff541880: Status 404 returned error can't find the container with id b3695c431bf15fd0e09e00efe76f0b5be8315502c6149b71776e9bd3ff541880
Oct 30 01:25:58.600311 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:58.600285    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77" event={"ID":"a1156317-e78b-4e13-a989-fe28b628cde1","Type":"ContainerStarted","Data":"b3695c431bf15fd0e09e00efe76f0b5be8315502c6149b71776e9bd3ff541880"}
Oct 30 01:25:58.601734 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:58.601547    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"
Oct 30 01:25:58.601734 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:58.601581    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2" event={"ID":"2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca","Type":"ContainerDied","Data":"72567505f05713a6752dd09b6d1e788c03bc71a80b37aa5a81e524a456cfe776"}
Oct 30 01:25:58.601734 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:58.601637    6099 scope.go:122] "RemoveContainer" containerID="4847907fbfb9326367fc14657c0215056ea37a52cbdc03d808cddfb31db65231"
Oct 30 01:25:58.652320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:58.652294    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"]
Oct 30 01:25:58.664088 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:58.664063    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8e24a5-openshift-default-549fd6595c-2stl2"]
Oct 30 01:25:59.609809 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:59.609696    6099 generic.go:356] "Generic (PLEG): container finished" podID="a1156317-e78b-4e13-a989-fe28b628cde1" containerID="5a551d5766a2ee643c05197a5d52d41b776881e6ca76abcb9555863b08584507" exitCode=0
Oct 30 01:25:59.609809 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:25:59.609779    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77" event={"ID":"a1156317-e78b-4e13-a989-fe28b628cde1","Type":"ContainerDied","Data":"5a551d5766a2ee643c05197a5d52d41b776881e6ca76abcb9555863b08584507"}
Oct 30 01:26:00.260469 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:00.260436    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca" path="/var/lib/kubelet/pods/2eeb488e-9da7-4e8d-bfaf-0e8a4aa4d8ca/volumes"
Oct 30 01:26:00.756068 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:00.756041    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77"
Oct 30 01:26:00.815585 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:00.815554    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1156317-e78b-4e13-a989-fe28b628cde1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1156317-e78b-4e13-a989-fe28b628cde1-maas-api-openshift-ca\") pod \"a1156317-e78b-4e13-a989-fe28b628cde1\" (UID: \"a1156317-e78b-4e13-a989-fe28b628cde1\") "
Oct 30 01:26:00.815781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:00.815593    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1156317-e78b-4e13-a989-fe28b628cde1-kube-api-access-pdchp\" (UniqueName: \"kubernetes.io/projected/a1156317-e78b-4e13-a989-fe28b628cde1-kube-api-access-pdchp\") pod \"a1156317-e78b-4e13-a989-fe28b628cde1\" (UID: \"a1156317-e78b-4e13-a989-fe28b628cde1\") "
Oct 30 01:26:00.815781 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:00.815757    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1156317-e78b-4e13-a989-fe28b628cde1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1156317-e78b-4e13-a989-fe28b628cde1-maas-api-xks-ca\") pod \"a1156317-e78b-4e13-a989-fe28b628cde1\" (UID: \"a1156317-e78b-4e13-a989-fe28b628cde1\") "
Oct 30 01:26:00.816026 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:00.815998    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1156317-e78b-4e13-a989-fe28b628cde1-maas-api-openshift-ca" pod "a1156317-e78b-4e13-a989-fe28b628cde1" (UID: "a1156317-e78b-4e13-a989-fe28b628cde1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:00.816257 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:00.816233    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1156317-e78b-4e13-a989-fe28b628cde1-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:00.817809 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:00.817783    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1156317-e78b-4e13-a989-fe28b628cde1-kube-api-access-pdchp" pod "a1156317-e78b-4e13-a989-fe28b628cde1" (UID: "a1156317-e78b-4e13-a989-fe28b628cde1"). InnerVolumeSpecName "kube-api-access-pdchp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:00.818255 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:00.818229    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1156317-e78b-4e13-a989-fe28b628cde1-maas-api-xks-ca" pod "a1156317-e78b-4e13-a989-fe28b628cde1" (UID: "a1156317-e78b-4e13-a989-fe28b628cde1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:26:00.917823 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:00.917761    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdchp\" (UniqueName: \"kubernetes.io/projected/a1156317-e78b-4e13-a989-fe28b628cde1-kube-api-access-pdchp\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:00.917823 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:00.917785    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1156317-e78b-4e13-a989-fe28b628cde1-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:01.627003 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:01.626967    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77"
Oct 30 01:26:01.627224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:01.626997    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77" event={"ID":"a1156317-e78b-4e13-a989-fe28b628cde1","Type":"ContainerDied","Data":"b3695c431bf15fd0e09e00efe76f0b5be8315502c6149b71776e9bd3ff541880"}
Oct 30 01:26:01.627224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:01.627057    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3695c431bf15fd0e09e00efe76f0b5be8315502c6149b71776e9bd3ff541880"
Oct 30 01:26:02.720527 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:02.720487    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"]
Oct 30 01:26:02.761086 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:02.761054    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"]
Oct 30 01:26:06.666406 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:06.666374    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:26:08.354019 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:08.353984    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h"]
Oct 30 01:26:09.190549 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:09.190515    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88"]
Oct 30 01:26:09.202391 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:09.202366    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-beb8d5-nbp88"]
Oct 30 01:26:10.260628 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:10.260594    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79665e1a-c51c-4623-8c73-f505c73bf38b" path="/var/lib/kubelet/pods/79665e1a-c51c-4623-8c73-f505c73bf38b/volumes"
Oct 30 01:26:10.770881 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:10.770845    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk"]
Oct 30 01:26:10.771348 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:10.771333    6099 cpu_manager.go:414] "containerMap: removing container" podUID="a1156317-e78b-4e13-a989-fe28b628cde1" containerName="revoke-keys"
Oct 30 01:26:10.771419 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:10.771349    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1156317-e78b-4e13-a989-fe28b628cde1" containerName="revoke-keys"
Oct 30 01:26:10.771463 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:10.771426    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1156317-e78b-4e13-a989-fe28b628cde1" containerName="revoke-keys"
Oct 30 01:26:10.774718 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:10.774687    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk"
Oct 30 01:26:10.800585 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:10.800538    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk"]
Oct 30 01:26:10.803824 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:10.803798    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:26:10.810391 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:10.810369    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:26:10.810499 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:10.810455    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:26:10.810499 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:10.810455    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:26:10.922084 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:10.922051    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3420de78-6b4a-4fa8-b78d-b9129cabd93f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee3ff2c1-2qqhk\" (UID: \"3420de78-6b4a-4fa8-b78d-b9129cabd93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk"
Oct 30 01:26:10.922253 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:10.922106    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4dp7\" (UniqueName: \"kubernetes.io/projected/3420de78-6b4a-4fa8-b78d-b9129cabd93f-kube-api-access-w4dp7\") pod \"maas-api-revoke-sub-ee3ff2c1-2qqhk\" (UID: \"3420de78-6b4a-4fa8-b78d-b9129cabd93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk"
Oct 30 01:26:10.922387 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:10.922296    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3420de78-6b4a-4fa8-b78d-b9129cabd93f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee3ff2c1-2qqhk\" (UID: \"3420de78-6b4a-4fa8-b78d-b9129cabd93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk"
Oct 30 01:26:11.023680 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:11.023609    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3420de78-6b4a-4fa8-b78d-b9129cabd93f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee3ff2c1-2qqhk\" (UID: \"3420de78-6b4a-4fa8-b78d-b9129cabd93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk"
Oct 30 01:26:11.023680 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:11.023664    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3420de78-6b4a-4fa8-b78d-b9129cabd93f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee3ff2c1-2qqhk\" (UID: \"3420de78-6b4a-4fa8-b78d-b9129cabd93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk"
Oct 30 01:26:11.023863 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:11.023725    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4dp7\" (UniqueName: \"kubernetes.io/projected/3420de78-6b4a-4fa8-b78d-b9129cabd93f-kube-api-access-w4dp7\") pod \"maas-api-revoke-sub-ee3ff2c1-2qqhk\" (UID: \"3420de78-6b4a-4fa8-b78d-b9129cabd93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk"
Oct 30 01:26:11.024407 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:11.024383    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3420de78-6b4a-4fa8-b78d-b9129cabd93f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee3ff2c1-2qqhk\" (UID: \"3420de78-6b4a-4fa8-b78d-b9129cabd93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk"
Oct 30 01:26:11.026023 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:11.026003    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3420de78-6b4a-4fa8-b78d-b9129cabd93f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee3ff2c1-2qqhk\" (UID: \"3420de78-6b4a-4fa8-b78d-b9129cabd93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk"
Oct 30 01:26:11.046747 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:11.046695    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4dp7\" (UniqueName: \"kubernetes.io/projected/3420de78-6b4a-4fa8-b78d-b9129cabd93f-kube-api-access-w4dp7\") pod \"maas-api-revoke-sub-ee3ff2c1-2qqhk\" (UID: \"3420de78-6b4a-4fa8-b78d-b9129cabd93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk"
Oct 30 01:26:11.088193 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:11.088171    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk"
Oct 30 01:26:11.384544 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:11.384519    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk"]
Oct 30 01:26:11.386166 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:26:11.386130    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3420de78_6b4a_4fa8_b78d_b9129cabd93f.slice/crio-5f0e4bd55740f7c8be9562e87d06f7a6ef9f1c9e7e7ec0c568455c960dda153f WatchSource:0}: Error finding container 5f0e4bd55740f7c8be9562e87d06f7a6ef9f1c9e7e7ec0c568455c960dda153f: Status 404 returned error can't find the container with id 5f0e4bd55740f7c8be9562e87d06f7a6ef9f1c9e7e7ec0c568455c960dda153f
Oct 30 01:26:11.692565 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:11.692533    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk" event={"ID":"3420de78-6b4a-4fa8-b78d-b9129cabd93f","Type":"ContainerStarted","Data":"22dc558f479e628c6fab9700cc411df8ef6c462c6322f1e6cbc1be81146c20c6"}
Oct 30 01:26:11.692716 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:11.692610    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk" event={"ID":"3420de78-6b4a-4fa8-b78d-b9129cabd93f","Type":"ContainerStarted","Data":"5f0e4bd55740f7c8be9562e87d06f7a6ef9f1c9e7e7ec0c568455c960dda153f"}
Oct 30 01:26:11.732066 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:11.731999    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk" podStartSLOduration=1.731985043 podStartE2EDuration="1.731985043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:11.730856856 +0000 UTC m=+3065.587764611" watchObservedRunningTime="2026-10-30 01:26:11.731985043 +0000 UTC m=+3065.588892798"
Oct 30 01:26:12.701522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:12.701487    6099 generic.go:356] "Generic (PLEG): container finished" podID="3420de78-6b4a-4fa8-b78d-b9129cabd93f" containerID="22dc558f479e628c6fab9700cc411df8ef6c462c6322f1e6cbc1be81146c20c6" exitCode=0
Oct 30 01:26:12.701953 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:12.701567    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk" event={"ID":"3420de78-6b4a-4fa8-b78d-b9129cabd93f","Type":"ContainerDied","Data":"22dc558f479e628c6fab9700cc411df8ef6c462c6322f1e6cbc1be81146c20c6"}
Oct 30 01:26:13.355238 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:13.355200    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h" podUID="97c0665f-6367-4539-a67d-3cc6b53fec21" containerName="payload-pre-processing" containerID="cri-o://081da4fcde248d7be0ca0b482aa8832947123a9d3bf2d794b11f0fa3275a1437" gracePeriod=25
Oct 30 01:26:13.871271 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:13.871245    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk"
Oct 30 01:26:13.954746 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:13.954687    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3420de78-6b4a-4fa8-b78d-b9129cabd93f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3420de78-6b4a-4fa8-b78d-b9129cabd93f-maas-api-xks-ca\") pod \"3420de78-6b4a-4fa8-b78d-b9129cabd93f\" (UID: \"3420de78-6b4a-4fa8-b78d-b9129cabd93f\") "
Oct 30 01:26:13.954941 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:13.954806    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3420de78-6b4a-4fa8-b78d-b9129cabd93f-kube-api-access-w4dp7\" (UniqueName: \"kubernetes.io/projected/3420de78-6b4a-4fa8-b78d-b9129cabd93f-kube-api-access-w4dp7\") pod \"3420de78-6b4a-4fa8-b78d-b9129cabd93f\" (UID: \"3420de78-6b4a-4fa8-b78d-b9129cabd93f\") "
Oct 30 01:26:13.954941 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:13.954838    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3420de78-6b4a-4fa8-b78d-b9129cabd93f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3420de78-6b4a-4fa8-b78d-b9129cabd93f-maas-api-openshift-ca\") pod \"3420de78-6b4a-4fa8-b78d-b9129cabd93f\" (UID: \"3420de78-6b4a-4fa8-b78d-b9129cabd93f\") "
Oct 30 01:26:13.955263 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:13.955238    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3420de78-6b4a-4fa8-b78d-b9129cabd93f-maas-api-openshift-ca" pod "3420de78-6b4a-4fa8-b78d-b9129cabd93f" (UID: "3420de78-6b4a-4fa8-b78d-b9129cabd93f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:13.956865 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:13.956833    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3420de78-6b4a-4fa8-b78d-b9129cabd93f-maas-api-xks-ca" pod "3420de78-6b4a-4fa8-b78d-b9129cabd93f" (UID: "3420de78-6b4a-4fa8-b78d-b9129cabd93f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:26:13.956965 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:13.956872    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3420de78-6b4a-4fa8-b78d-b9129cabd93f-kube-api-access-w4dp7" pod "3420de78-6b4a-4fa8-b78d-b9129cabd93f" (UID: "3420de78-6b4a-4fa8-b78d-b9129cabd93f"). InnerVolumeSpecName "kube-api-access-w4dp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:14.056595 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:14.056562    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3420de78-6b4a-4fa8-b78d-b9129cabd93f-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:14.056595 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:14.056590    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4dp7\" (UniqueName: \"kubernetes.io/projected/3420de78-6b4a-4fa8-b78d-b9129cabd93f-kube-api-access-w4dp7\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:14.056595 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:14.056601    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3420de78-6b4a-4fa8-b78d-b9129cabd93f-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:14.716466 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:14.716434    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk"
Oct 30 01:26:14.716643 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:14.716477    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk" event={"ID":"3420de78-6b4a-4fa8-b78d-b9129cabd93f","Type":"ContainerDied","Data":"5f0e4bd55740f7c8be9562e87d06f7a6ef9f1c9e7e7ec0c568455c960dda153f"}
Oct 30 01:26:14.716643 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:14.716531    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f0e4bd55740f7c8be9562e87d06f7a6ef9f1c9e7e7ec0c568455c960dda153f"
Oct 30 01:26:16.043448 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:16.043413    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"]
Oct 30 01:26:16.043911 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:16.043694    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh" podUID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerName="istio-proxy" containerID="cri-o://ea02c4366bf188a3dd41c2f3ca0bbabc289fe40ef39602a7f995477e11fd72c1" gracePeriod=30
Oct 30 01:26:16.628984 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:26:16.628933    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:26:17.740833 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:17.740793    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main" containerID="cri-o://92d3ce385dd6fdf638aef991a6ea7819b4cb9edeb9041bce39f47c345778d075" gracePeriod=45
Oct 30 01:26:17.780087 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:17.780055    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main" containerID="cri-o://37f62e9ea5065c3d44e742f3ab0b92bde8bf4924a7960b1121b7ab8c9a6e39d3" gracePeriod=45
Oct 30 01:26:18.037394 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.037364    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 01:26:18.056192 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.056165    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 01:26:18.201306 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.201270    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-home\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-home\") pod \"80fa6588-19e4-4f9a-8291-12775c169de4\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") "
Oct 30 01:26:18.201521 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.201362    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-tmp-dir\") pod \"80fa6588-19e4-4f9a-8291-12775c169de4\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") "
Oct 30 01:26:18.201521 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.201413    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-dshm\") pod \"c3662307-8768-40a7-91f4-1d36cc1b41b9\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") "
Oct 30 01:26:18.201521 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.201436    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3662307-8768-40a7-91f4-1d36cc1b41b9-kube-api-access-q2tmk\" (UniqueName: \"kubernetes.io/projected/c3662307-8768-40a7-91f4-1d36cc1b41b9-kube-api-access-q2tmk\") pod \"c3662307-8768-40a7-91f4-1d36cc1b41b9\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") "
Oct 30 01:26:18.201521 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.201464    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-model-cache\") pod \"80fa6588-19e4-4f9a-8291-12775c169de4\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") "
Oct 30 01:26:18.201521 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.201487    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-home\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-home\") pod \"c3662307-8768-40a7-91f4-1d36cc1b41b9\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") "
Oct 30 01:26:18.201521 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.201509    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80fa6588-19e4-4f9a-8291-12775c169de4-kube-api-access-8n6hn\" (UniqueName: \"kubernetes.io/projected/80fa6588-19e4-4f9a-8291-12775c169de4-kube-api-access-8n6hn\") pod \"80fa6588-19e4-4f9a-8291-12775c169de4\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") "
Oct 30 01:26:18.201892 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.201538    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-home" pod "80fa6588-19e4-4f9a-8291-12775c169de4" (UID: "80fa6588-19e4-4f9a-8291-12775c169de4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:18.201892 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.201591    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-model-cache\") pod \"c3662307-8768-40a7-91f4-1d36cc1b41b9\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") "
Oct 30 01:26:18.201892 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.201632    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-tmp-dir" pod "80fa6588-19e4-4f9a-8291-12775c169de4" (UID: "80fa6588-19e4-4f9a-8291-12775c169de4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:18.201892 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.201660    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80fa6588-19e4-4f9a-8291-12775c169de4-tls-certs\" (UniqueName: \"kubernetes.io/secret/80fa6588-19e4-4f9a-8291-12775c169de4-tls-certs\") pod \"80fa6588-19e4-4f9a-8291-12775c169de4\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") "
Oct 30 01:26:18.201892 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.201751    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-tmp-dir\") pod \"c3662307-8768-40a7-91f4-1d36cc1b41b9\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") "
Oct 30 01:26:18.201892 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.201787    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3662307-8768-40a7-91f4-1d36cc1b41b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/c3662307-8768-40a7-91f4-1d36cc1b41b9-tls-certs\") pod \"c3662307-8768-40a7-91f4-1d36cc1b41b9\" (UID: \"c3662307-8768-40a7-91f4-1d36cc1b41b9\") "
Oct 30 01:26:18.201892 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.201839    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-dshm\") pod \"80fa6588-19e4-4f9a-8291-12775c169de4\" (UID: \"80fa6588-19e4-4f9a-8291-12775c169de4\") "
Oct 30 01:26:18.202275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.201785    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-model-cache" pod "80fa6588-19e4-4f9a-8291-12775c169de4" (UID: "80fa6588-19e4-4f9a-8291-12775c169de4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:18.202275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.202190    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-model-cache" pod "c3662307-8768-40a7-91f4-1d36cc1b41b9" (UID: "c3662307-8768-40a7-91f4-1d36cc1b41b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:18.202275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.202204    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-tmp-dir" pod "c3662307-8768-40a7-91f4-1d36cc1b41b9" (UID: "c3662307-8768-40a7-91f4-1d36cc1b41b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:18.202275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.202215    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-home" pod "c3662307-8768-40a7-91f4-1d36cc1b41b9" (UID: "c3662307-8768-40a7-91f4-1d36cc1b41b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:18.202482 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.202283    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:18.202482 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.202329    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:18.204453 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.204426    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80fa6588-19e4-4f9a-8291-12775c169de4-kube-api-access-8n6hn" pod "80fa6588-19e4-4f9a-8291-12775c169de4" (UID: "80fa6588-19e4-4f9a-8291-12775c169de4"). InnerVolumeSpecName "kube-api-access-8n6hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:18.204600 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.204460    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3662307-8768-40a7-91f4-1d36cc1b41b9-kube-api-access-q2tmk" pod "c3662307-8768-40a7-91f4-1d36cc1b41b9" (UID: "c3662307-8768-40a7-91f4-1d36cc1b41b9"). InnerVolumeSpecName "kube-api-access-q2tmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:18.204600 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.204535    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-dshm" pod "c3662307-8768-40a7-91f4-1d36cc1b41b9" (UID: "c3662307-8768-40a7-91f4-1d36cc1b41b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:18.204600 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.204547    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80fa6588-19e4-4f9a-8291-12775c169de4-tls-certs" pod "80fa6588-19e4-4f9a-8291-12775c169de4" (UID: "80fa6588-19e4-4f9a-8291-12775c169de4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:26:18.204745 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.204647    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3662307-8768-40a7-91f4-1d36cc1b41b9-tls-certs" pod "c3662307-8768-40a7-91f4-1d36cc1b41b9" (UID: "c3662307-8768-40a7-91f4-1d36cc1b41b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:26:18.204799 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.204758    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-dshm" pod "80fa6588-19e4-4f9a-8291-12775c169de4" (UID: "80fa6588-19e4-4f9a-8291-12775c169de4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:18.303675 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.303640    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:18.303675 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.303667    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80fa6588-19e4-4f9a-8291-12775c169de4-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:18.303675 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.303676    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:18.303903 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.303683    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3662307-8768-40a7-91f4-1d36cc1b41b9-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:18.303903 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.303691    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:18.303903 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.303699    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:18.303903 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.303734    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2tmk\" (UniqueName: \"kubernetes.io/projected/c3662307-8768-40a7-91f4-1d36cc1b41b9-kube-api-access-q2tmk\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:18.303903 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.303750    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80fa6588-19e4-4f9a-8291-12775c169de4-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:18.303903 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.303762    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3662307-8768-40a7-91f4-1d36cc1b41b9-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:18.303903 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.303772    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n6hn\" (UniqueName: \"kubernetes.io/projected/80fa6588-19e4-4f9a-8291-12775c169de4-kube-api-access-8n6hn\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:18.437968 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.437929    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h" podUID="97c0665f-6367-4539-a67d-3cc6b53fec21" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:26:18.744716 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.744675    6099 generic.go:356] "Generic (PLEG): container finished" podID="80fa6588-19e4-4f9a-8291-12775c169de4" containerID="37f62e9ea5065c3d44e742f3ab0b92bde8bf4924a7960b1121b7ab8c9a6e39d3" exitCode=0
Oct 30 01:26:18.745241 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.744735    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"
Oct 30 01:26:18.745241 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.744734    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" event={"ID":"80fa6588-19e4-4f9a-8291-12775c169de4","Type":"ContainerDied","Data":"37f62e9ea5065c3d44e742f3ab0b92bde8bf4924a7960b1121b7ab8c9a6e39d3"}
Oct 30 01:26:18.745241 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.744868    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q" event={"ID":"80fa6588-19e4-4f9a-8291-12775c169de4","Type":"ContainerDied","Data":"d87f854b47c317f61f821e24dfcc30c6836d0e9993088b58762d44ce0076c022"}
Oct 30 01:26:18.745241 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.744886    6099 scope.go:122] "RemoveContainer" containerID="37f62e9ea5065c3d44e742f3ab0b92bde8bf4924a7960b1121b7ab8c9a6e39d3"
Oct 30 01:26:18.746293 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.746228    6099 generic.go:356] "Generic (PLEG): container finished" podID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerID="92d3ce385dd6fdf638aef991a6ea7819b4cb9edeb9041bce39f47c345778d075" exitCode=0
Oct 30 01:26:18.746293 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.746282    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"
Oct 30 01:26:18.746426 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.746318    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" event={"ID":"c3662307-8768-40a7-91f4-1d36cc1b41b9","Type":"ContainerDied","Data":"92d3ce385dd6fdf638aef991a6ea7819b4cb9edeb9041bce39f47c345778d075"}
Oct 30 01:26:18.746426 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.746362    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t" event={"ID":"c3662307-8768-40a7-91f4-1d36cc1b41b9","Type":"ContainerDied","Data":"1591b2c398995f1edacd057f14ea7f9f1f973a93c05524a2f413f1836e638442"}
Oct 30 01:26:18.755858 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.755840    6099 scope.go:122] "RemoveContainer" containerID="37f62e9ea5065c3d44e742f3ab0b92bde8bf4924a7960b1121b7ab8c9a6e39d3"
Oct 30 01:26:18.756181 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:26:18.756153    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f62e9ea5065c3d44e742f3ab0b92bde8bf4924a7960b1121b7ab8c9a6e39d3\": container with ID starting with 37f62e9ea5065c3d44e742f3ab0b92bde8bf4924a7960b1121b7ab8c9a6e39d3 not found: ID does not exist" containerID="37f62e9ea5065c3d44e742f3ab0b92bde8bf4924a7960b1121b7ab8c9a6e39d3"
Oct 30 01:26:18.756289 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.756188    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f62e9ea5065c3d44e742f3ab0b92bde8bf4924a7960b1121b7ab8c9a6e39d3"} err="failed to get container status \"37f62e9ea5065c3d44e742f3ab0b92bde8bf4924a7960b1121b7ab8c9a6e39d3\": rpc error: code = NotFound desc = could not find container \"37f62e9ea5065c3d44e742f3ab0b92bde8bf4924a7960b1121b7ab8c9a6e39d3\": container with ID starting with 37f62e9ea5065c3d44e742f3ab0b92bde8bf4924a7960b1121b7ab8c9a6e39d3 not found: ID does not exist"
Oct 30 01:26:18.756289 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.756212    6099 scope.go:122] "RemoveContainer" containerID="92d3ce385dd6fdf638aef991a6ea7819b4cb9edeb9041bce39f47c345778d075"
Oct 30 01:26:18.770511 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.770488    6099 scope.go:122] "RemoveContainer" containerID="92d3ce385dd6fdf638aef991a6ea7819b4cb9edeb9041bce39f47c345778d075"
Oct 30 01:26:18.770826 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:26:18.770805    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92d3ce385dd6fdf638aef991a6ea7819b4cb9edeb9041bce39f47c345778d075\": container with ID starting with 92d3ce385dd6fdf638aef991a6ea7819b4cb9edeb9041bce39f47c345778d075 not found: ID does not exist" containerID="92d3ce385dd6fdf638aef991a6ea7819b4cb9edeb9041bce39f47c345778d075"
Oct 30 01:26:18.770904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.770833    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92d3ce385dd6fdf638aef991a6ea7819b4cb9edeb9041bce39f47c345778d075"} err="failed to get container status \"92d3ce385dd6fdf638aef991a6ea7819b4cb9edeb9041bce39f47c345778d075\": rpc error: code = NotFound desc = could not find container \"92d3ce385dd6fdf638aef991a6ea7819b4cb9edeb9041bce39f47c345778d075\": container with ID starting with 92d3ce385dd6fdf638aef991a6ea7819b4cb9edeb9041bce39f47c345778d075 not found: ID does not exist"
Oct 30 01:26:18.796827 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.796804    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"]
Oct 30 01:26:18.801028 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.801005    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/premium-w5-89a676-kserve-5f558786f4-9lf6q"]
Oct 30 01:26:18.832951 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.832930    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"]
Oct 30 01:26:18.837315 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:18.837296    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-89a676/distinct-2-w5-89a676-kserve-6fbff7b454-nbr2t"]
Oct 30 01:26:19.704893 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:19.704861    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:26:20.260525 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:20.260494    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" path="/var/lib/kubelet/pods/80fa6588-19e4-4f9a-8291-12775c169de4/volumes"
Oct 30 01:26:20.260941 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:20.260930    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" path="/var/lib/kubelet/pods/c3662307-8768-40a7-91f4-1d36cc1b41b9/volumes"
Oct 30 01:26:21.300679 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.300656    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 01:26:21.436595 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.436518    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-credential-socket\") pod \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") "
Oct 30 01:26:21.436595 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.436555    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-data\") pod \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") "
Oct 30 01:26:21.436595 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.436595    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-workload-socket\") pod \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") "
Oct 30 01:26:21.436901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.436634    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-podinfo\") pod \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") "
Oct 30 01:26:21.436901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.436657    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-kube-api-access-bv2jh\" (UniqueName: \"kubernetes.io/projected/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-kube-api-access-bv2jh\") pod \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") "
Oct 30 01:26:21.436901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.436675    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istiod-ca-cert\") pod \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") "
Oct 30 01:26:21.436901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.436784    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-envoy\") pod \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") "
Oct 30 01:26:21.436901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.436824    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-token\" (UniqueName: \"kubernetes.io/projected/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-token\") pod \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") "
Oct 30 01:26:21.436901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.436878    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-workload-certs\") pod \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\" (UID: \"6de29f4b-4f9e-4e67-a43a-f414dc775ee3\") "
Oct 30 01:26:21.437217 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.436950    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-data" pod "6de29f4b-4f9e-4e67-a43a-f414dc775ee3" (UID: "6de29f4b-4f9e-4e67-a43a-f414dc775ee3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:21.437217 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.436962    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-credential-socket" pod "6de29f4b-4f9e-4e67-a43a-f414dc775ee3" (UID: "6de29f4b-4f9e-4e67-a43a-f414dc775ee3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:21.437217 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.437029    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-workload-socket" pod "6de29f4b-4f9e-4e67-a43a-f414dc775ee3" (UID: "6de29f4b-4f9e-4e67-a43a-f414dc775ee3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:21.437414 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.437247    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istiod-ca-cert" pod "6de29f4b-4f9e-4e67-a43a-f414dc775ee3" (UID: "6de29f4b-4f9e-4e67-a43a-f414dc775ee3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:21.437779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.437751    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-workload-certs" pod "6de29f4b-4f9e-4e67-a43a-f414dc775ee3" (UID: "6de29f4b-4f9e-4e67-a43a-f414dc775ee3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:21.437913 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.437805    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:21.438095 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.438059    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:21.438095 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.438080    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:21.438095 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.438093    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:21.439217 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.439193    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-podinfo" pod "6de29f4b-4f9e-4e67-a43a-f414dc775ee3" (UID: "6de29f4b-4f9e-4e67-a43a-f414dc775ee3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:26:21.439217 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.439212    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-token" pod "6de29f4b-4f9e-4e67-a43a-f414dc775ee3" (UID: "6de29f4b-4f9e-4e67-a43a-f414dc775ee3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:21.439414 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.439400    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-envoy" pod "6de29f4b-4f9e-4e67-a43a-f414dc775ee3" (UID: "6de29f4b-4f9e-4e67-a43a-f414dc775ee3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:21.439612 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.439597    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-kube-api-access-bv2jh" pod "6de29f4b-4f9e-4e67-a43a-f414dc775ee3" (UID: "6de29f4b-4f9e-4e67-a43a-f414dc775ee3"). InnerVolumeSpecName "kube-api-access-bv2jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:21.539547 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.539515    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:21.539547 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.539542    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:21.539757 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.539557    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:21.539757 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.539568    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:21.539757 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.539579    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv2jh\" (UniqueName: \"kubernetes.io/projected/6de29f4b-4f9e-4e67-a43a-f414dc775ee3-kube-api-access-bv2jh\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:21.767683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.767604    6099 generic.go:356] "Generic (PLEG): container finished" podID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerID="ea02c4366bf188a3dd41c2f3ca0bbabc289fe40ef39602a7f995477e11fd72c1" exitCode=0
Oct 30 01:26:21.767683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.767663    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"
Oct 30 01:26:21.767940 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.767687    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh" event={"ID":"6de29f4b-4f9e-4e67-a43a-f414dc775ee3","Type":"ContainerDied","Data":"ea02c4366bf188a3dd41c2f3ca0bbabc289fe40ef39602a7f995477e11fd72c1"}
Oct 30 01:26:21.767940 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.767773    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh" event={"ID":"6de29f4b-4f9e-4e67-a43a-f414dc775ee3","Type":"ContainerDied","Data":"ff516610eb85a249e75efdfeca4659b76c25de44de589dd98a615ad84d667dd6"}
Oct 30 01:26:21.767940 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.767804    6099 scope.go:122] "RemoveContainer" containerID="ea02c4366bf188a3dd41c2f3ca0bbabc289fe40ef39602a7f995477e11fd72c1"
Oct 30 01:26:21.778655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.778636    6099 scope.go:122] "RemoveContainer" containerID="4747ec3dae7bc350f37088743008ae26ea8c4f5860526badd4f50b04ae133c0b"
Oct 30 01:26:21.787802 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.787783    6099 scope.go:122] "RemoveContainer" containerID="ea02c4366bf188a3dd41c2f3ca0bbabc289fe40ef39602a7f995477e11fd72c1"
Oct 30 01:26:21.788040 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:26:21.788024    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea02c4366bf188a3dd41c2f3ca0bbabc289fe40ef39602a7f995477e11fd72c1\": container with ID starting with ea02c4366bf188a3dd41c2f3ca0bbabc289fe40ef39602a7f995477e11fd72c1 not found: ID does not exist" containerID="ea02c4366bf188a3dd41c2f3ca0bbabc289fe40ef39602a7f995477e11fd72c1"
Oct 30 01:26:21.788106 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.788047    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea02c4366bf188a3dd41c2f3ca0bbabc289fe40ef39602a7f995477e11fd72c1"} err="failed to get container status \"ea02c4366bf188a3dd41c2f3ca0bbabc289fe40ef39602a7f995477e11fd72c1\": rpc error: code = NotFound desc = could not find container \"ea02c4366bf188a3dd41c2f3ca0bbabc289fe40ef39602a7f995477e11fd72c1\": container with ID starting with ea02c4366bf188a3dd41c2f3ca0bbabc289fe40ef39602a7f995477e11fd72c1 not found: ID does not exist"
Oct 30 01:26:21.788106 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.788063    6099 scope.go:122] "RemoveContainer" containerID="4747ec3dae7bc350f37088743008ae26ea8c4f5860526badd4f50b04ae133c0b"
Oct 30 01:26:21.788275 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:26:21.788262    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4747ec3dae7bc350f37088743008ae26ea8c4f5860526badd4f50b04ae133c0b\": container with ID starting with 4747ec3dae7bc350f37088743008ae26ea8c4f5860526badd4f50b04ae133c0b not found: ID does not exist" containerID="4747ec3dae7bc350f37088743008ae26ea8c4f5860526badd4f50b04ae133c0b"
Oct 30 01:26:21.788321 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.788279    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4747ec3dae7bc350f37088743008ae26ea8c4f5860526badd4f50b04ae133c0b"} err="failed to get container status \"4747ec3dae7bc350f37088743008ae26ea8c4f5860526badd4f50b04ae133c0b\": rpc error: code = NotFound desc = could not find container \"4747ec3dae7bc350f37088743008ae26ea8c4f5860526badd4f50b04ae133c0b\": container with ID starting with 4747ec3dae7bc350f37088743008ae26ea8c4f5860526badd4f50b04ae133c0b not found: ID does not exist"
Oct 30 01:26:21.818206 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.818182    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"]
Oct 30 01:26:21.830257 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:21.830236    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-beb8d5-openshift-default-fc65648b7-z4hdh"]
Oct 30 01:26:22.260446 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:22.260410    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" path="/var/lib/kubelet/pods/6de29f4b-4f9e-4e67-a43a-f414dc775ee3/volumes"
Oct 30 01:26:26.541185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:26.541150    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77"]
Oct 30 01:26:26.553053 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:26.553023    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-89a676-84x77"]
Oct 30 01:26:28.241015 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:28.240973    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:26:28.260383 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:28.260346    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1156317-e78b-4e13-a989-fe28b628cde1" path="/var/lib/kubelet/pods/a1156317-e78b-4e13-a989-fe28b628cde1/volumes"
Oct 30 01:26:28.437394 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:28.437354    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h" podUID="97c0665f-6367-4539-a67d-3cc6b53fec21" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:26:33.603869 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:33.603845    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h"
Oct 30 01:26:33.666043 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:33.665953    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97c0665f-6367-4539-a67d-3cc6b53fec21-kube-api-access-bl5dq\" (UniqueName: \"kubernetes.io/projected/97c0665f-6367-4539-a67d-3cc6b53fec21-kube-api-access-bl5dq\") pod \"97c0665f-6367-4539-a67d-3cc6b53fec21\" (UID: \"97c0665f-6367-4539-a67d-3cc6b53fec21\") "
Oct 30 01:26:33.666223 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:33.666106    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97c0665f-6367-4539-a67d-3cc6b53fec21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c0665f-6367-4539-a67d-3cc6b53fec21-plugins-config-volume\") pod \"97c0665f-6367-4539-a67d-3cc6b53fec21\" (UID: \"97c0665f-6367-4539-a67d-3cc6b53fec21\") "
Oct 30 01:26:33.666759 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:33.666699    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c0665f-6367-4539-a67d-3cc6b53fec21-plugins-config-volume" pod "97c0665f-6367-4539-a67d-3cc6b53fec21" (UID: "97c0665f-6367-4539-a67d-3cc6b53fec21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:33.679737 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:33.669455    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c0665f-6367-4539-a67d-3cc6b53fec21-kube-api-access-bl5dq" pod "97c0665f-6367-4539-a67d-3cc6b53fec21" (UID: "97c0665f-6367-4539-a67d-3cc6b53fec21"). InnerVolumeSpecName "kube-api-access-bl5dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:33.685079 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:33.685049    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"]
Oct 30 01:26:33.685426 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:33.685268    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s" podUID="f5d7156f-8120-4ab7-83ae-f7ab80171e94" containerName="istio-proxy" containerID="cri-o://bb92c0feca22e00fd02e56c5a70c866a75ca036759acd41c0e303861b42a5270" gracePeriod=30
Oct 30 01:26:33.767238 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:33.767208    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c0665f-6367-4539-a67d-3cc6b53fec21-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:33.767238 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:33.767234    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl5dq\" (UniqueName: \"kubernetes.io/projected/97c0665f-6367-4539-a67d-3cc6b53fec21-kube-api-access-bl5dq\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:33.860491 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:33.860455    6099 generic.go:356] "Generic (PLEG): container finished" podID="97c0665f-6367-4539-a67d-3cc6b53fec21" containerID="081da4fcde248d7be0ca0b482aa8832947123a9d3bf2d794b11f0fa3275a1437" exitCode=0
Oct 30 01:26:33.860695 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:33.860497    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h"
Oct 30 01:26:33.860695 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:33.860538    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h" event={"ID":"97c0665f-6367-4539-a67d-3cc6b53fec21","Type":"ContainerDied","Data":"081da4fcde248d7be0ca0b482aa8832947123a9d3bf2d794b11f0fa3275a1437"}
Oct 30 01:26:33.860695 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:33.860598    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h" event={"ID":"97c0665f-6367-4539-a67d-3cc6b53fec21","Type":"ContainerDied","Data":"75a1e8bc46511cbcdb7facbd59881bb90fdc6bb779f5af5f1f1eb336bc2bc964"}
Oct 30 01:26:33.860695 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:33.860627    6099 scope.go:122] "RemoveContainer" containerID="081da4fcde248d7be0ca0b482aa8832947123a9d3bf2d794b11f0fa3275a1437"
Oct 30 01:26:33.871342 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:33.871320    6099 scope.go:122] "RemoveContainer" containerID="081da4fcde248d7be0ca0b482aa8832947123a9d3bf2d794b11f0fa3275a1437"
Oct 30 01:26:33.871610 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:26:33.871580    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"081da4fcde248d7be0ca0b482aa8832947123a9d3bf2d794b11f0fa3275a1437\": container with ID starting with 081da4fcde248d7be0ca0b482aa8832947123a9d3bf2d794b11f0fa3275a1437 not found: ID does not exist" containerID="081da4fcde248d7be0ca0b482aa8832947123a9d3bf2d794b11f0fa3275a1437"
Oct 30 01:26:33.871721 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:33.871611    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081da4fcde248d7be0ca0b482aa8832947123a9d3bf2d794b11f0fa3275a1437"} err="failed to get container status \"081da4fcde248d7be0ca0b482aa8832947123a9d3bf2d794b11f0fa3275a1437\": rpc error: code = NotFound desc = could not find container \"081da4fcde248d7be0ca0b482aa8832947123a9d3bf2d794b11f0fa3275a1437\": container with ID starting with 081da4fcde248d7be0ca0b482aa8832947123a9d3bf2d794b11f0fa3275a1437 not found: ID does not exist"
Oct 30 01:26:33.909593 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:33.909569    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h"]
Oct 30 01:26:33.916299 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:33.916250    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-89a676-6d9c994f55-5vk7h"]
Oct 30 01:26:34.260663 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:34.260589    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97c0665f-6367-4539-a67d-3cc6b53fec21" path="/var/lib/kubelet/pods/97c0665f-6367-4539-a67d-3cc6b53fec21/volumes"
Oct 30 01:26:38.895893 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:38.895849    6099 generic.go:356] "Generic (PLEG): container finished" podID="f5d7156f-8120-4ab7-83ae-f7ab80171e94" containerID="bb92c0feca22e00fd02e56c5a70c866a75ca036759acd41c0e303861b42a5270" exitCode=0
Oct 30 01:26:38.896283 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:38.895921    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s" event={"ID":"f5d7156f-8120-4ab7-83ae-f7ab80171e94","Type":"ContainerDied","Data":"bb92c0feca22e00fd02e56c5a70c866a75ca036759acd41c0e303861b42a5270"}
Oct 30 01:26:38.946058 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:38.946023    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 01:26:39.019050 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.018959    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-token\" (UniqueName: \"kubernetes.io/projected/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-token\") pod \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") "
Oct 30 01:26:39.019050 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.019011    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-workload-certs\") pod \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") "
Oct 30 01:26:39.019050 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.019046    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-workload-socket\") pod \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") "
Oct 30 01:26:39.019361 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.019100    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-podinfo\") pod \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") "
Oct 30 01:26:39.019361 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.019128    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-envoy\") pod \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") "
Oct 30 01:26:39.019361 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.019170    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-credential-socket\") pod \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") "
Oct 30 01:26:39.019361 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.019200    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istiod-ca-cert\") pod \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") "
Oct 30 01:26:39.019361 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.019279    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-data\") pod \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") "
Oct 30 01:26:39.019361 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.019315    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d7156f-8120-4ab7-83ae-f7ab80171e94-kube-api-access-68djq\" (UniqueName: \"kubernetes.io/projected/f5d7156f-8120-4ab7-83ae-f7ab80171e94-kube-api-access-68djq\") pod \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\" (UID: \"f5d7156f-8120-4ab7-83ae-f7ab80171e94\") "
Oct 30 01:26:39.020724 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.020147    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-credential-socket" pod "f5d7156f-8120-4ab7-83ae-f7ab80171e94" (UID: "f5d7156f-8120-4ab7-83ae-f7ab80171e94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:39.022275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.021512    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istiod-ca-cert" pod "f5d7156f-8120-4ab7-83ae-f7ab80171e94" (UID: "f5d7156f-8120-4ab7-83ae-f7ab80171e94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:39.022275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.021843    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-data" pod "f5d7156f-8120-4ab7-83ae-f7ab80171e94" (UID: "f5d7156f-8120-4ab7-83ae-f7ab80171e94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:39.022275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.022033    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-workload-certs" pod "f5d7156f-8120-4ab7-83ae-f7ab80171e94" (UID: "f5d7156f-8120-4ab7-83ae-f7ab80171e94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:39.022275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.022236    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-workload-socket" pod "f5d7156f-8120-4ab7-83ae-f7ab80171e94" (UID: "f5d7156f-8120-4ab7-83ae-f7ab80171e94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:39.025004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.024968    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d7156f-8120-4ab7-83ae-f7ab80171e94-kube-api-access-68djq" pod "f5d7156f-8120-4ab7-83ae-f7ab80171e94" (UID: "f5d7156f-8120-4ab7-83ae-f7ab80171e94"). InnerVolumeSpecName "kube-api-access-68djq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:39.025795 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.025767    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-podinfo" pod "f5d7156f-8120-4ab7-83ae-f7ab80171e94" (UID: "f5d7156f-8120-4ab7-83ae-f7ab80171e94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:26:39.026027 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.025993    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-token" pod "f5d7156f-8120-4ab7-83ae-f7ab80171e94" (UID: "f5d7156f-8120-4ab7-83ae-f7ab80171e94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:39.026591 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.026193    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-envoy" pod "f5d7156f-8120-4ab7-83ae-f7ab80171e94" (UID: "f5d7156f-8120-4ab7-83ae-f7ab80171e94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:39.120760 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.120728    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:39.120760 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.120759    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:39.121040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.120769    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:39.121040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.120778    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68djq\" (UniqueName: \"kubernetes.io/projected/f5d7156f-8120-4ab7-83ae-f7ab80171e94-kube-api-access-68djq\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:39.121040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.120786    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:39.121040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.120794    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:39.121040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.120802    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:39.121040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.120812    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:39.121040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.120821    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d7156f-8120-4ab7-83ae-f7ab80171e94-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:26:39.904759 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.904717    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s" event={"ID":"f5d7156f-8120-4ab7-83ae-f7ab80171e94","Type":"ContainerDied","Data":"ff9995f562d6cc6a3b4eb4074e10548aad8eae122d7c7f39bc2f082c45c8620d"}
Oct 30 01:26:39.905295 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.904802    6099 scope.go:122] "RemoveContainer" containerID="bb92c0feca22e00fd02e56c5a70c866a75ca036759acd41c0e303861b42a5270"
Oct 30 01:26:39.905295 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.904728    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"
Oct 30 01:26:39.968603 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.968570    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"]
Oct 30 01:26:39.972670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:39.972640    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-89a676-openshift-default-6bdc5cc785-fcx7s"]
Oct 30 01:26:40.260110 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:40.260030    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5d7156f-8120-4ab7-83ae-f7ab80171e94" path="/var/lib/kubelet/pods/f5d7156f-8120-4ab7-83ae-f7ab80171e94/volumes"
Oct 30 01:26:46.628739 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:26:46.628679    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:26:57.799490 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.799451    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"]
Oct 30 01:26:57.800108 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800087    6099 cpu_manager.go:414] "containerMap: removing container" podUID="f5d7156f-8120-4ab7-83ae-f7ab80171e94" containerName="istio-proxy"
Oct 30 01:26:57.800196 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800110    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5d7156f-8120-4ab7-83ae-f7ab80171e94" containerName="istio-proxy"
Oct 30 01:26:57.800196 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800139    6099 cpu_manager.go:414] "containerMap: removing container" podUID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerName="istio-proxy"
Oct 30 01:26:57.800196 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800147    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerName="istio-proxy"
Oct 30 01:26:57.800196 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800163    6099 cpu_manager.go:414] "containerMap: removing container" podUID="3420de78-6b4a-4fa8-b78d-b9129cabd93f" containerName="revoke-keys"
Oct 30 01:26:57.800196 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800171    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3420de78-6b4a-4fa8-b78d-b9129cabd93f" containerName="revoke-keys"
Oct 30 01:26:57.800196 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800184    6099 cpu_manager.go:414] "containerMap: removing container" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main"
Oct 30 01:26:57.800196 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800191    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main"
Oct 30 01:26:57.800573 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800212    6099 cpu_manager.go:414] "containerMap: removing container" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main"
Oct 30 01:26:57.800573 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800220    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main"
Oct 30 01:26:57.800573 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800235    6099 cpu_manager.go:414] "containerMap: removing container" podUID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerName="istio-proxy"
Oct 30 01:26:57.800573 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800242    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerName="istio-proxy"
Oct 30 01:26:57.800573 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800257    6099 cpu_manager.go:414] "containerMap: removing container" podUID="97c0665f-6367-4539-a67d-3cc6b53fec21" containerName="payload-pre-processing"
Oct 30 01:26:57.800573 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800264    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c0665f-6367-4539-a67d-3cc6b53fec21" containerName="payload-pre-processing"
Oct 30 01:26:57.800573 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800380    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerName="istio-proxy"
Oct 30 01:26:57.800573 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800395    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c0665f-6367-4539-a67d-3cc6b53fec21" containerName="payload-pre-processing"
Oct 30 01:26:57.800573 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800410    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5d7156f-8120-4ab7-83ae-f7ab80171e94" containerName="istio-proxy"
Oct 30 01:26:57.800573 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800423    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3420de78-6b4a-4fa8-b78d-b9129cabd93f" containerName="revoke-keys"
Oct 30 01:26:57.800573 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800434    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3662307-8768-40a7-91f4-1d36cc1b41b9" containerName="main"
Oct 30 01:26:57.800573 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.800447    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="80fa6588-19e4-4f9a-8291-12775c169de4" containerName="main"
Oct 30 01:26:57.805688 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.805666    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.836059 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.836035    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"]
Oct 30 01:26:57.893490 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.893452    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e1681f-ad08-4174-86a2-261d28114fb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.893490 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.893492    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e1681f-ad08-4174-86a2-261d28114fb8-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.893744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.893599    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e1681f-ad08-4174-86a2-261d28114fb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.893744 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.893701    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e1681f-ad08-4174-86a2-261d28114fb8-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.893848 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.893746    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e1681f-ad08-4174-86a2-261d28114fb8-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.893848 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.893790    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e1681f-ad08-4174-86a2-261d28114fb8-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.893848 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.893832    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e1681f-ad08-4174-86a2-261d28114fb8-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.893996 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.893878    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jsfh\" (UniqueName: \"kubernetes.io/projected/53e1681f-ad08-4174-86a2-261d28114fb8-kube-api-access-2jsfh\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.893996 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.893951    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e1681f-ad08-4174-86a2-261d28114fb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.994786 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.994755    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e1681f-ad08-4174-86a2-261d28114fb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.994960 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.994803    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e1681f-ad08-4174-86a2-261d28114fb8-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.994960 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.994831    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e1681f-ad08-4174-86a2-261d28114fb8-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.994960 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.994882    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e1681f-ad08-4174-86a2-261d28114fb8-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.994960 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.994912    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e1681f-ad08-4174-86a2-261d28114fb8-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.994960 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.994939    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jsfh\" (UniqueName: \"kubernetes.io/projected/53e1681f-ad08-4174-86a2-261d28114fb8-kube-api-access-2jsfh\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.995254 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.994970    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e1681f-ad08-4174-86a2-261d28114fb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.995254 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.995014    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e1681f-ad08-4174-86a2-261d28114fb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.995254 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.995038    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e1681f-ad08-4174-86a2-261d28114fb8-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.995464 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.995409    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e1681f-ad08-4174-86a2-261d28114fb8-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.995600 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.995491    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e1681f-ad08-4174-86a2-261d28114fb8-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.995600 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.995498    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e1681f-ad08-4174-86a2-261d28114fb8-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.995772 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.995657    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e1681f-ad08-4174-86a2-261d28114fb8-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.995833 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.995789    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e1681f-ad08-4174-86a2-261d28114fb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.997417 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.997397    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e1681f-ad08-4174-86a2-261d28114fb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:57.997519 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:57.997398    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e1681f-ad08-4174-86a2-261d28114fb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:58.017796 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:58.017765    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jsfh\" (UniqueName: \"kubernetes.io/projected/53e1681f-ad08-4174-86a2-261d28114fb8-kube-api-access-2jsfh\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:58.017796 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:58.017778    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e1681f-ad08-4174-86a2-261d28114fb8-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2\" (UID: \"53e1681f-ad08-4174-86a2-261d28114fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:58.122936 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:58.122861    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:58.489083 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:58.489057    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"]
Oct 30 01:26:58.491391 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:26:58.491353    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e1681f_ad08_4174_86a2_261d28114fb8.slice/crio-ee6256c742b4c171af9510993184f565db30672d8f42d9bc39c8bb814f28a818 WatchSource:0}: Error finding container ee6256c742b4c171af9510993184f565db30672d8f42d9bc39c8bb814f28a818: Status 404 returned error can't find the container with id ee6256c742b4c171af9510993184f565db30672d8f42d9bc39c8bb814f28a818
Oct 30 01:26:58.493519 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:58.493482    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:26:58.493603 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:58.493551    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:26:58.493603 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:58.493582    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:26:59.015250 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:59.015208    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2" event={"ID":"53e1681f-ad08-4174-86a2-261d28114fb8","Type":"ContainerStarted","Data":"a035ff2a71beb76fb34f28a5a04a5e5835dacad42bbc76b572ba5612c214482f"}
Oct 30 01:26:59.015655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:59.015274    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2" event={"ID":"53e1681f-ad08-4174-86a2-261d28114fb8","Type":"ContainerStarted","Data":"ee6256c742b4c171af9510993184f565db30672d8f42d9bc39c8bb814f28a818"}
Oct 30 01:26:59.123321 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:59.123285    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:26:59.124888 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:59.124861    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" start-of-body=
Oct 30 01:26:59.125053 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:26:59.124929    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2" podUID="53e1681f-ad08-4174-86a2-261d28114fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused"
Oct 30 01:27:00.123221 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:00.123185    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" start-of-body=
Oct 30 01:27:00.123623 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:00.123232    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2" podUID="53e1681f-ad08-4174-86a2-261d28114fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused"
Oct 30 01:27:01.124130 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:01.124091    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" start-of-body=
Oct 30 01:27:01.124641 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:01.124143    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2" podUID="53e1681f-ad08-4174-86a2-261d28114fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused"
Oct 30 01:27:01.756382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:01.756331    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2" podStartSLOduration=4.756316817 podStartE2EDuration="4.756316817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:59.082044461 +0000 UTC m=+3112.938952217" watchObservedRunningTime="2026-10-30 01:27:01.756316817 +0000 UTC m=+3115.613224569"
Oct 30 01:27:01.758044 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:01.758014    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc"]
Oct 30 01:27:01.758559 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:01.758539    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de29f4b-4f9e-4e67-a43a-f414dc775ee3" containerName="istio-proxy"
Oct 30 01:27:01.761862 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:01.761841    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc"
Oct 30 01:27:01.788280 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:01.782320    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc"]
Oct 30 01:27:01.792144 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:01.792110    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:27:01.792440 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:01.792360    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:27:01.792575 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:01.792394    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:27:01.792683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:01.792446    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:27:01.932158 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:01.932126    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86683fc9-cc0c-4228-9e0d-9ae63b326713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-h4gdc\" (UID: \"86683fc9-cc0c-4228-9e0d-9ae63b326713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc"
Oct 30 01:27:01.932337 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:01.932191    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86683fc9-cc0c-4228-9e0d-9ae63b326713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-h4gdc\" (UID: \"86683fc9-cc0c-4228-9e0d-9ae63b326713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc"
Oct 30 01:27:01.932337 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:01.932282    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s86w7\" (UniqueName: \"kubernetes.io/projected/86683fc9-cc0c-4228-9e0d-9ae63b326713-kube-api-access-s86w7\") pod \"maas-api-revoke-sub-85237e45-h4gdc\" (UID: \"86683fc9-cc0c-4228-9e0d-9ae63b326713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc"
Oct 30 01:27:02.033397 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:02.033313    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s86w7\" (UniqueName: \"kubernetes.io/projected/86683fc9-cc0c-4228-9e0d-9ae63b326713-kube-api-access-s86w7\") pod \"maas-api-revoke-sub-85237e45-h4gdc\" (UID: \"86683fc9-cc0c-4228-9e0d-9ae63b326713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc"
Oct 30 01:27:02.033397 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:02.033394    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86683fc9-cc0c-4228-9e0d-9ae63b326713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-h4gdc\" (UID: \"86683fc9-cc0c-4228-9e0d-9ae63b326713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc"
Oct 30 01:27:02.033604 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:02.033436    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86683fc9-cc0c-4228-9e0d-9ae63b326713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-h4gdc\" (UID: \"86683fc9-cc0c-4228-9e0d-9ae63b326713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc"
Oct 30 01:27:02.034119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:02.034095    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86683fc9-cc0c-4228-9e0d-9ae63b326713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-h4gdc\" (UID: \"86683fc9-cc0c-4228-9e0d-9ae63b326713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc"
Oct 30 01:27:02.035826 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:02.035802    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86683fc9-cc0c-4228-9e0d-9ae63b326713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-h4gdc\" (UID: \"86683fc9-cc0c-4228-9e0d-9ae63b326713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc"
Oct 30 01:27:02.056444 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:02.056421    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s86w7\" (UniqueName: \"kubernetes.io/projected/86683fc9-cc0c-4228-9e0d-9ae63b326713-kube-api-access-s86w7\") pod \"maas-api-revoke-sub-85237e45-h4gdc\" (UID: \"86683fc9-cc0c-4228-9e0d-9ae63b326713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc"
Oct 30 01:27:02.102933 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:02.102905    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc"
Oct 30 01:27:02.123485 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:02.123456    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" start-of-body=
Oct 30 01:27:02.123611 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:02.123511    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2" podUID="53e1681f-ad08-4174-86a2-261d28114fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused"
Oct 30 01:27:02.398685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:02.398657    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc"]
Oct 30 01:27:02.399681 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:27:02.399651    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86683fc9_cc0c_4228_9e0d_9ae63b326713.slice/crio-0b30f345d0459fb5caa7a32e119bb7bfb1d12de8a6f8274259b99f792a1e78c9 WatchSource:0}: Error finding container 0b30f345d0459fb5caa7a32e119bb7bfb1d12de8a6f8274259b99f792a1e78c9: Status 404 returned error can't find the container with id 0b30f345d0459fb5caa7a32e119bb7bfb1d12de8a6f8274259b99f792a1e78c9
Oct 30 01:27:03.040312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:03.040269    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc" event={"ID":"86683fc9-cc0c-4228-9e0d-9ae63b326713","Type":"ContainerStarted","Data":"e8b619f56bf3ec520ac61093ce89573278ce6b146e46c1c10597ed1d917c6688"}
Oct 30 01:27:03.040507 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:03.040334    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc" event={"ID":"86683fc9-cc0c-4228-9e0d-9ae63b326713","Type":"ContainerStarted","Data":"0b30f345d0459fb5caa7a32e119bb7bfb1d12de8a6f8274259b99f792a1e78c9"}
Oct 30 01:27:03.089810 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:03.084150    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc" podStartSLOduration=2.084132975 podStartE2EDuration="2.084132975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:03.08308803 +0000 UTC m=+3116.939995809" watchObservedRunningTime="2026-10-30 01:27:03.084132975 +0000 UTC m=+3116.941040731"
Oct 30 01:27:03.124020 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:03.123988    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" start-of-body=
Oct 30 01:27:03.124174 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:03.124036    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2" podUID="53e1681f-ad08-4174-86a2-261d28114fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused"
Oct 30 01:27:04.047498 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:04.047462    6099 generic.go:356] "Generic (PLEG): container finished" podID="86683fc9-cc0c-4228-9e0d-9ae63b326713" containerID="e8b619f56bf3ec520ac61093ce89573278ce6b146e46c1c10597ed1d917c6688" exitCode=0
Oct 30 01:27:04.047910 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:04.047506    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc" event={"ID":"86683fc9-cc0c-4228-9e0d-9ae63b326713","Type":"ContainerDied","Data":"e8b619f56bf3ec520ac61093ce89573278ce6b146e46c1c10597ed1d917c6688"}
Oct 30 01:27:04.124018 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:04.123985    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" start-of-body=
Oct 30 01:27:04.124207 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:04.124044    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2" podUID="53e1681f-ad08-4174-86a2-261d28114fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused"
Oct 30 01:27:05.124151 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:05.124120    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" start-of-body=
Oct 30 01:27:05.124609 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:05.124168    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2" podUID="53e1681f-ad08-4174-86a2-261d28114fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused"
Oct 30 01:27:05.198365 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:05.198339    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc"
Oct 30 01:27:05.263424 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:05.263382    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86683fc9-cc0c-4228-9e0d-9ae63b326713-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86683fc9-cc0c-4228-9e0d-9ae63b326713-maas-api-xks-ca\") pod \"86683fc9-cc0c-4228-9e0d-9ae63b326713\" (UID: \"86683fc9-cc0c-4228-9e0d-9ae63b326713\") "
Oct 30 01:27:05.263608 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:05.263480    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86683fc9-cc0c-4228-9e0d-9ae63b326713-kube-api-access-s86w7\" (UniqueName: \"kubernetes.io/projected/86683fc9-cc0c-4228-9e0d-9ae63b326713-kube-api-access-s86w7\") pod \"86683fc9-cc0c-4228-9e0d-9ae63b326713\" (UID: \"86683fc9-cc0c-4228-9e0d-9ae63b326713\") "
Oct 30 01:27:05.263608 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:05.263505    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86683fc9-cc0c-4228-9e0d-9ae63b326713-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86683fc9-cc0c-4228-9e0d-9ae63b326713-maas-api-openshift-ca\") pod \"86683fc9-cc0c-4228-9e0d-9ae63b326713\" (UID: \"86683fc9-cc0c-4228-9e0d-9ae63b326713\") "
Oct 30 01:27:05.263931 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:05.263902    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86683fc9-cc0c-4228-9e0d-9ae63b326713-maas-api-openshift-ca" pod "86683fc9-cc0c-4228-9e0d-9ae63b326713" (UID: "86683fc9-cc0c-4228-9e0d-9ae63b326713"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:05.265518 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:05.265494    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86683fc9-cc0c-4228-9e0d-9ae63b326713-kube-api-access-s86w7" pod "86683fc9-cc0c-4228-9e0d-9ae63b326713" (UID: "86683fc9-cc0c-4228-9e0d-9ae63b326713"). InnerVolumeSpecName "kube-api-access-s86w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:05.265941 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:05.265916    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86683fc9-cc0c-4228-9e0d-9ae63b326713-maas-api-xks-ca" pod "86683fc9-cc0c-4228-9e0d-9ae63b326713" (UID: "86683fc9-cc0c-4228-9e0d-9ae63b326713"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:27:05.365088 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:05.365012    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s86w7\" (UniqueName: \"kubernetes.io/projected/86683fc9-cc0c-4228-9e0d-9ae63b326713-kube-api-access-s86w7\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:27:05.365088 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:05.365037    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86683fc9-cc0c-4228-9e0d-9ae63b326713-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:27:05.365088 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:05.365047    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86683fc9-cc0c-4228-9e0d-9ae63b326713-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:27:06.062240 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:06.062206    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc"
Oct 30 01:27:06.062419 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:06.062205    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc" event={"ID":"86683fc9-cc0c-4228-9e0d-9ae63b326713","Type":"ContainerDied","Data":"0b30f345d0459fb5caa7a32e119bb7bfb1d12de8a6f8274259b99f792a1e78c9"}
Oct 30 01:27:06.062419 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:06.062333    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b30f345d0459fb5caa7a32e119bb7bfb1d12de8a6f8274259b99f792a1e78c9"
Oct 30 01:27:07.124653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:07.124618    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:27:07.125152 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:07.124721    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2" podUID="53e1681f-ad08-4174-86a2-261d28114fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:08.165893 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:08.165850    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:27:08.166344 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:08.165944    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2" podUID="53e1681f-ad08-4174-86a2-261d28114fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:08.790796 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:08.790766    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:27:08.791060 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:08.791037    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:27:08.791811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:08.791789    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2"
Oct 30 01:27:15.375794 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:15.375761    6099 scope.go:122] "RemoveContainer" containerID="78997ac4350d149f69ab67bc3078822ab537ac0a5cefb52fb1c182658c9b31d0"
Oct 30 01:27:15.385160 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:15.385138    6099 scope.go:122] "RemoveContainer" containerID="6f208a5ac28415e9799eee4e1731c978f732e8dcc34442bfe6f4109a7361e627"
Oct 30 01:27:16.629327 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:27:16.629290    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:27:35.029751 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:35.029718    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj"]
Oct 30 01:27:35.041423 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:35.041396    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a0c9803-5wpzj"]
Oct 30 01:27:36.259698 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:36.259667    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f79a646-0479-486f-8b0b-dda8e7ed0ac4" path="/var/lib/kubelet/pods/8f79a646-0479-486f-8b0b-dda8e7ed0ac4/volumes"
Oct 30 01:27:43.305450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:43.305412    6099 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal" podUID="f8c3a9edbd8cbf03175beab8c4db7242" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 01:27:43.931871 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:43.931836    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:27:44.027740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:44.027691    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt"]
Oct 30 01:27:44.040133 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:44.040101    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw"]
Oct 30 01:27:44.041552 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:44.041532    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcad322b-99nxt"]
Oct 30 01:27:44.042818 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:44.042798    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6086b2-4x5jw"]
Oct 30 01:27:44.260174 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:44.260092    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad0363d5-bc18-418f-8974-4d783829a6f4" path="/var/lib/kubelet/pods/ad0363d5-bc18-418f-8974-4d783829a6f4/volumes"
Oct 30 01:27:44.260518 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:44.260503    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e774fee5-42fc-42c9-b02c-6b1cad0f80a8" path="/var/lib/kubelet/pods/e774fee5-42fc-42c9-b02c-6b1cad0f80a8/volumes"
Oct 30 01:27:46.628470 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:27:46.628432    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:27:49.857029 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:49.856979    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548"]
Oct 30 01:27:49.857451 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:49.857436    6099 cpu_manager.go:414] "containerMap: removing container" podUID="86683fc9-cc0c-4228-9e0d-9ae63b326713" containerName="revoke-keys"
Oct 30 01:27:49.857497 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:49.857452    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86683fc9-cc0c-4228-9e0d-9ae63b326713" containerName="revoke-keys"
Oct 30 01:27:49.857553 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:49.857542    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="86683fc9-cc0c-4228-9e0d-9ae63b326713" containerName="revoke-keys"
Oct 30 01:27:49.861897 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:49.861873    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548"
Oct 30 01:27:49.877964 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:49.877942    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:27:49.878109 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:49.878053    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:27:49.889898 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:49.889865    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:27:49.889898 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:49.889890    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:27:49.891628 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:49.891604    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548"]
Oct 30 01:27:49.979108 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:49.979079    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c5351f-6289-41cc-8a25-3dab37ca2913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-4s548\" (UID: \"a8c5351f-6289-41cc-8a25-3dab37ca2913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548"
Oct 30 01:27:49.979302 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:49.979227    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c5351f-6289-41cc-8a25-3dab37ca2913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-4s548\" (UID: \"a8c5351f-6289-41cc-8a25-3dab37ca2913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548"
Oct 30 01:27:49.979383 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:49.979301    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz6df\" (UniqueName: \"kubernetes.io/projected/a8c5351f-6289-41cc-8a25-3dab37ca2913-kube-api-access-lz6df\") pod \"maas-api-revoke-sub-c1e271a9-4s548\" (UID: \"a8c5351f-6289-41cc-8a25-3dab37ca2913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548"
Oct 30 01:27:50.080121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:50.080090    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c5351f-6289-41cc-8a25-3dab37ca2913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-4s548\" (UID: \"a8c5351f-6289-41cc-8a25-3dab37ca2913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548"
Oct 30 01:27:50.080310 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:50.080215    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c5351f-6289-41cc-8a25-3dab37ca2913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-4s548\" (UID: \"a8c5351f-6289-41cc-8a25-3dab37ca2913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548"
Oct 30 01:27:50.080310 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:50.080258    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz6df\" (UniqueName: \"kubernetes.io/projected/a8c5351f-6289-41cc-8a25-3dab37ca2913-kube-api-access-lz6df\") pod \"maas-api-revoke-sub-c1e271a9-4s548\" (UID: \"a8c5351f-6289-41cc-8a25-3dab37ca2913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548"
Oct 30 01:27:50.080926 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:50.080900    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c5351f-6289-41cc-8a25-3dab37ca2913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-4s548\" (UID: \"a8c5351f-6289-41cc-8a25-3dab37ca2913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548"
Oct 30 01:27:50.082473 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:50.082452    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c5351f-6289-41cc-8a25-3dab37ca2913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-4s548\" (UID: \"a8c5351f-6289-41cc-8a25-3dab37ca2913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548"
Oct 30 01:27:50.103617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:50.103592    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz6df\" (UniqueName: \"kubernetes.io/projected/a8c5351f-6289-41cc-8a25-3dab37ca2913-kube-api-access-lz6df\") pod \"maas-api-revoke-sub-c1e271a9-4s548\" (UID: \"a8c5351f-6289-41cc-8a25-3dab37ca2913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548"
Oct 30 01:27:50.174518 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:50.174445    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548"
Oct 30 01:27:50.472805 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:50.472778    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548"]
Oct 30 01:27:50.475114 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:27:50.475085    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c5351f_6289_41cc_8a25_3dab37ca2913.slice/crio-d30c2b06c0e4b9bc732612de402bf7dc7c309df5ca6b8a820958ed388e2b6204 WatchSource:0}: Error finding container d30c2b06c0e4b9bc732612de402bf7dc7c309df5ca6b8a820958ed388e2b6204: Status 404 returned error can't find the container with id d30c2b06c0e4b9bc732612de402bf7dc7c309df5ca6b8a820958ed388e2b6204
Oct 30 01:27:51.325216 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:51.325181    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548" event={"ID":"a8c5351f-6289-41cc-8a25-3dab37ca2913","Type":"ContainerStarted","Data":"458e51e5a0c4796dcf449754600464105287d9b7b6d864b5b1bc189f92158ce9"}
Oct 30 01:27:51.325594 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:51.325235    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548" event={"ID":"a8c5351f-6289-41cc-8a25-3dab37ca2913","Type":"ContainerStarted","Data":"d30c2b06c0e4b9bc732612de402bf7dc7c309df5ca6b8a820958ed388e2b6204"}
Oct 30 01:27:51.364236 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:51.364193    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548" podStartSLOduration=2.36417916 podStartE2EDuration="2.36417916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:51.363944269 +0000 UTC m=+3165.220852028" watchObservedRunningTime="2026-10-30 01:27:51.36417916 +0000 UTC m=+3165.221086915"
Oct 30 01:27:52.332241 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:52.332206    6099 generic.go:356] "Generic (PLEG): container finished" podID="a8c5351f-6289-41cc-8a25-3dab37ca2913" containerID="458e51e5a0c4796dcf449754600464105287d9b7b6d864b5b1bc189f92158ce9" exitCode=0
Oct 30 01:27:52.332626 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:52.332286    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548" event={"ID":"a8c5351f-6289-41cc-8a25-3dab37ca2913","Type":"ContainerDied","Data":"458e51e5a0c4796dcf449754600464105287d9b7b6d864b5b1bc189f92158ce9"}
Oct 30 01:27:53.478059 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:53.478034    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548"
Oct 30 01:27:53.615100 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:53.615016    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c5351f-6289-41cc-8a25-3dab37ca2913-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c5351f-6289-41cc-8a25-3dab37ca2913-maas-api-openshift-ca\") pod \"a8c5351f-6289-41cc-8a25-3dab37ca2913\" (UID: \"a8c5351f-6289-41cc-8a25-3dab37ca2913\") "
Oct 30 01:27:53.615100 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:53.615088    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c5351f-6289-41cc-8a25-3dab37ca2913-kube-api-access-lz6df\" (UniqueName: \"kubernetes.io/projected/a8c5351f-6289-41cc-8a25-3dab37ca2913-kube-api-access-lz6df\") pod \"a8c5351f-6289-41cc-8a25-3dab37ca2913\" (UID: \"a8c5351f-6289-41cc-8a25-3dab37ca2913\") "
Oct 30 01:27:53.615359 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:53.615171    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8c5351f-6289-41cc-8a25-3dab37ca2913-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c5351f-6289-41cc-8a25-3dab37ca2913-maas-api-xks-ca\") pod \"a8c5351f-6289-41cc-8a25-3dab37ca2913\" (UID: \"a8c5351f-6289-41cc-8a25-3dab37ca2913\") "
Oct 30 01:27:53.615453 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:53.615428    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c5351f-6289-41cc-8a25-3dab37ca2913-maas-api-openshift-ca" pod "a8c5351f-6289-41cc-8a25-3dab37ca2913" (UID: "a8c5351f-6289-41cc-8a25-3dab37ca2913"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:53.617275 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:53.617248    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c5351f-6289-41cc-8a25-3dab37ca2913-maas-api-xks-ca" pod "a8c5351f-6289-41cc-8a25-3dab37ca2913" (UID: "a8c5351f-6289-41cc-8a25-3dab37ca2913"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:27:53.617379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:53.617332    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c5351f-6289-41cc-8a25-3dab37ca2913-kube-api-access-lz6df" pod "a8c5351f-6289-41cc-8a25-3dab37ca2913" (UID: "a8c5351f-6289-41cc-8a25-3dab37ca2913"). InnerVolumeSpecName "kube-api-access-lz6df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:53.716119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:53.716080    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c5351f-6289-41cc-8a25-3dab37ca2913-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:27:53.716119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:53.716118    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c5351f-6289-41cc-8a25-3dab37ca2913-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:27:53.716278 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:53.716133    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz6df\" (UniqueName: \"kubernetes.io/projected/a8c5351f-6289-41cc-8a25-3dab37ca2913-kube-api-access-lz6df\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:27:54.344798 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:54.344768    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548"
Oct 30 01:27:54.344798 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:54.344780    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548" event={"ID":"a8c5351f-6289-41cc-8a25-3dab37ca2913","Type":"ContainerDied","Data":"d30c2b06c0e4b9bc732612de402bf7dc7c309df5ca6b8a820958ed388e2b6204"}
Oct 30 01:27:54.345009 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:27:54.344825    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d30c2b06c0e4b9bc732612de402bf7dc7c309df5ca6b8a820958ed388e2b6204"
Oct 30 01:28:01.110305 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:01.110274    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:28:05.455938 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:05.455901    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:28:10.881393 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:10.881353    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc"]
Oct 30 01:28:10.881972 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:10.881953    6099 cpu_manager.go:414] "containerMap: removing container" podUID="a8c5351f-6289-41cc-8a25-3dab37ca2913" containerName="revoke-keys"
Oct 30 01:28:10.882023 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:10.881974    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c5351f-6289-41cc-8a25-3dab37ca2913" containerName="revoke-keys"
Oct 30 01:28:10.882061 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:10.882042    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c5351f-6289-41cc-8a25-3dab37ca2913" containerName="revoke-keys"
Oct 30 01:28:10.885237 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:10.885212    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc"
Oct 30 01:28:10.899631 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:10.899607    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:28:10.911678 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:10.911650    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:28:10.911849 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:10.911689    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:28:10.911849 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:10.911746    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:28:10.914914 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:10.914877    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc"]
Oct 30 01:28:10.969236 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:10.969204    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz7bt\" (UniqueName: \"kubernetes.io/projected/98f86982-37a2-45e5-8028-2bbc099ca613-kube-api-access-rz7bt\") pod \"maas-api-revoke-sub-2e1f5b5f-4lbdc\" (UID: \"98f86982-37a2-45e5-8028-2bbc099ca613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc"
Oct 30 01:28:10.969236 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:10.969243    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f86982-37a2-45e5-8028-2bbc099ca613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4lbdc\" (UID: \"98f86982-37a2-45e5-8028-2bbc099ca613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc"
Oct 30 01:28:10.969458 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:10.969267    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f86982-37a2-45e5-8028-2bbc099ca613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4lbdc\" (UID: \"98f86982-37a2-45e5-8028-2bbc099ca613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc"
Oct 30 01:28:11.070874 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:11.070836    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz7bt\" (UniqueName: \"kubernetes.io/projected/98f86982-37a2-45e5-8028-2bbc099ca613-kube-api-access-rz7bt\") pod \"maas-api-revoke-sub-2e1f5b5f-4lbdc\" (UID: \"98f86982-37a2-45e5-8028-2bbc099ca613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc"
Oct 30 01:28:11.070874 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:11.070874    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f86982-37a2-45e5-8028-2bbc099ca613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4lbdc\" (UID: \"98f86982-37a2-45e5-8028-2bbc099ca613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc"
Oct 30 01:28:11.071145 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:11.070907    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f86982-37a2-45e5-8028-2bbc099ca613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4lbdc\" (UID: \"98f86982-37a2-45e5-8028-2bbc099ca613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc"
Oct 30 01:28:11.071579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:11.071557    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f86982-37a2-45e5-8028-2bbc099ca613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4lbdc\" (UID: \"98f86982-37a2-45e5-8028-2bbc099ca613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc"
Oct 30 01:28:11.073202 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:11.073174    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f86982-37a2-45e5-8028-2bbc099ca613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4lbdc\" (UID: \"98f86982-37a2-45e5-8028-2bbc099ca613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc"
Oct 30 01:28:11.093959 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:11.093936    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz7bt\" (UniqueName: \"kubernetes.io/projected/98f86982-37a2-45e5-8028-2bbc099ca613-kube-api-access-rz7bt\") pod \"maas-api-revoke-sub-2e1f5b5f-4lbdc\" (UID: \"98f86982-37a2-45e5-8028-2bbc099ca613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc"
Oct 30 01:28:11.198430 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:11.198357    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc"
Oct 30 01:28:11.505496 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:11.505021    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc"]
Oct 30 01:28:12.447938 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:12.447906    6099 generic.go:356] "Generic (PLEG): container finished" podID="98f86982-37a2-45e5-8028-2bbc099ca613" containerID="42c3c703dd4cf083a2a1392026bd7433cbcf77de041338037f3f91a618e124f0" exitCode=0
Oct 30 01:28:12.448341 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:12.447979    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc" event={"ID":"98f86982-37a2-45e5-8028-2bbc099ca613","Type":"ContainerDied","Data":"42c3c703dd4cf083a2a1392026bd7433cbcf77de041338037f3f91a618e124f0"}
Oct 30 01:28:12.448341 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:12.448032    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc" event={"ID":"98f86982-37a2-45e5-8028-2bbc099ca613","Type":"ContainerStarted","Data":"cc1607c32274e617678a483903499ea16a02afd95da5265517f4285bc2f07538"}
Oct 30 01:28:12.993483 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:12.993450    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"]
Oct 30 01:28:12.997332 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:12.997311    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.033911 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.033886    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"]
Oct 30 01:28:13.091058 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.091028    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77598a0-d6b3-421a-bd3d-5b13591ad686-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.091210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.091084    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77598a0-d6b3-421a-bd3d-5b13591ad686-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.091210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.091166    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77598a0-d6b3-421a-bd3d-5b13591ad686-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.091305 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.091226    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77598a0-d6b3-421a-bd3d-5b13591ad686-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.091305 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.091258    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77598a0-d6b3-421a-bd3d-5b13591ad686-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.091385 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.091314    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77598a0-d6b3-421a-bd3d-5b13591ad686-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.091385 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.091330    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77598a0-d6b3-421a-bd3d-5b13591ad686-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.091385 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.091352    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77598a0-d6b3-421a-bd3d-5b13591ad686-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.091385 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.091370    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lrx8\" (UniqueName: \"kubernetes.io/projected/d77598a0-d6b3-421a-bd3d-5b13591ad686-kube-api-access-6lrx8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.192227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.192194    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77598a0-d6b3-421a-bd3d-5b13591ad686-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.192370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.192253    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77598a0-d6b3-421a-bd3d-5b13591ad686-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.192370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.192275    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77598a0-d6b3-421a-bd3d-5b13591ad686-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.192370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.192306    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77598a0-d6b3-421a-bd3d-5b13591ad686-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.192370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.192330    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77598a0-d6b3-421a-bd3d-5b13591ad686-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.192370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.192359    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77598a0-d6b3-421a-bd3d-5b13591ad686-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.192668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.192383    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77598a0-d6b3-421a-bd3d-5b13591ad686-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.192668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.192419    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77598a0-d6b3-421a-bd3d-5b13591ad686-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.192668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.192446    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lrx8\" (UniqueName: \"kubernetes.io/projected/d77598a0-d6b3-421a-bd3d-5b13591ad686-kube-api-access-6lrx8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.192884 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.192821    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77598a0-d6b3-421a-bd3d-5b13591ad686-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.192884 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.192844    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77598a0-d6b3-421a-bd3d-5b13591ad686-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.192972 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.192894    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77598a0-d6b3-421a-bd3d-5b13591ad686-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.193090 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.193066    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77598a0-d6b3-421a-bd3d-5b13591ad686-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.193146 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.193122    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77598a0-d6b3-421a-bd3d-5b13591ad686-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.194648 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.194628    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77598a0-d6b3-421a-bd3d-5b13591ad686-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.195005 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.194985    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77598a0-d6b3-421a-bd3d-5b13591ad686-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.215631 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.215598    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77598a0-d6b3-421a-bd3d-5b13591ad686-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.215785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.215765    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lrx8\" (UniqueName: \"kubernetes.io/projected/d77598a0-d6b3-421a-bd3d-5b13591ad686-kube-api-access-6lrx8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-pc67m\" (UID: \"d77598a0-d6b3-421a-bd3d-5b13591ad686\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.311887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.311861    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:13.573104 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.573076    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc"
Oct 30 01:28:13.626851 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.626823    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"]
Oct 30 01:28:13.627726 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:28:13.627682    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77598a0_d6b3_421a_bd3d_5b13591ad686.slice/crio-7b16b4720006e742c4d78c918d6344585a648d27d34ebfca4c1861614e7eeede WatchSource:0}: Error finding container 7b16b4720006e742c4d78c918d6344585a648d27d34ebfca4c1861614e7eeede: Status 404 returned error can't find the container with id 7b16b4720006e742c4d78c918d6344585a648d27d34ebfca4c1861614e7eeede
Oct 30 01:28:13.630075 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.630042    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:28:13.630192 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.630106    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:28:13.630192 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.630134    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:28:13.696845 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.696824    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98f86982-37a2-45e5-8028-2bbc099ca613-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f86982-37a2-45e5-8028-2bbc099ca613-maas-api-openshift-ca\") pod \"98f86982-37a2-45e5-8028-2bbc099ca613\" (UID: \"98f86982-37a2-45e5-8028-2bbc099ca613\") "
Oct 30 01:28:13.696944 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.696873    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98f86982-37a2-45e5-8028-2bbc099ca613-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f86982-37a2-45e5-8028-2bbc099ca613-maas-api-xks-ca\") pod \"98f86982-37a2-45e5-8028-2bbc099ca613\" (UID: \"98f86982-37a2-45e5-8028-2bbc099ca613\") "
Oct 30 01:28:13.697004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.696985    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98f86982-37a2-45e5-8028-2bbc099ca613-kube-api-access-rz7bt\" (UniqueName: \"kubernetes.io/projected/98f86982-37a2-45e5-8028-2bbc099ca613-kube-api-access-rz7bt\") pod \"98f86982-37a2-45e5-8028-2bbc099ca613\" (UID: \"98f86982-37a2-45e5-8028-2bbc099ca613\") "
Oct 30 01:28:13.697228 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.697201    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f86982-37a2-45e5-8028-2bbc099ca613-maas-api-openshift-ca" pod "98f86982-37a2-45e5-8028-2bbc099ca613" (UID: "98f86982-37a2-45e5-8028-2bbc099ca613"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:13.698760 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.698738    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f86982-37a2-45e5-8028-2bbc099ca613-maas-api-xks-ca" pod "98f86982-37a2-45e5-8028-2bbc099ca613" (UID: "98f86982-37a2-45e5-8028-2bbc099ca613"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:28:13.699140 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.699119    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f86982-37a2-45e5-8028-2bbc099ca613-kube-api-access-rz7bt" pod "98f86982-37a2-45e5-8028-2bbc099ca613" (UID: "98f86982-37a2-45e5-8028-2bbc099ca613"). InnerVolumeSpecName "kube-api-access-rz7bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:13.798190 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.798150    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz7bt\" (UniqueName: \"kubernetes.io/projected/98f86982-37a2-45e5-8028-2bbc099ca613-kube-api-access-rz7bt\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:28:13.798190 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.798186    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f86982-37a2-45e5-8028-2bbc099ca613-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:28:13.798406 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:13.798202    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f86982-37a2-45e5-8028-2bbc099ca613-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:28:14.460770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:14.460661    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc"
Oct 30 01:28:14.460770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:14.460660    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc" event={"ID":"98f86982-37a2-45e5-8028-2bbc099ca613","Type":"ContainerDied","Data":"cc1607c32274e617678a483903499ea16a02afd95da5265517f4285bc2f07538"}
Oct 30 01:28:14.460770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:14.460765    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc1607c32274e617678a483903499ea16a02afd95da5265517f4285bc2f07538"
Oct 30 01:28:14.462433 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:14.462403    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m" event={"ID":"d77598a0-d6b3-421a-bd3d-5b13591ad686","Type":"ContainerStarted","Data":"82982ffab503603465155b2622dd0812fd281ff4a817ec843a3c277b1a50478c"}
Oct 30 01:28:14.462582 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:14.462456    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m" event={"ID":"d77598a0-d6b3-421a-bd3d-5b13591ad686","Type":"ContainerStarted","Data":"7b16b4720006e742c4d78c918d6344585a648d27d34ebfca4c1861614e7eeede"}
Oct 30 01:28:14.520606 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:14.520554    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m" podStartSLOduration=2.520540238 podStartE2EDuration="2.520540238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:14.519337061 +0000 UTC m=+3188.376244815" watchObservedRunningTime="2026-10-30 01:28:14.520540238 +0000 UTC m=+3188.377447990"
Oct 30 01:28:15.313026 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:15.312982    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:15.314604 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:15.314578    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" start-of-body=
Oct 30 01:28:15.314760 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:15.314639    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m" podUID="d77598a0-d6b3-421a-bd3d-5b13591ad686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused"
Oct 30 01:28:15.435667 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:15.435629    6099 scope.go:122] "RemoveContainer" containerID="ec53ff181874d2d99f502855dec3aa517c63bb474090e968581289dcb40e2fa0"
Oct 30 01:28:15.444964 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:15.444941    6099 scope.go:122] "RemoveContainer" containerID="d6dd90bdf3c8589d5b960c76a1446c688ffc896022e98adfbf4936b878d8a590"
Oct 30 01:28:15.452999 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:15.452976    6099 scope.go:122] "RemoveContainer" containerID="65aa649fd58c23d082c789c35989fd407fc73c59d199f3ec9efcddb9a228024e"
Oct 30 01:28:16.313185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:16.313151    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" start-of-body=
Oct 30 01:28:16.313599 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:16.313202    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m" podUID="d77598a0-d6b3-421a-bd3d-5b13591ad686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused"
Oct 30 01:28:16.628535 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:28:16.628442    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:28:17.313171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:17.313135    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" start-of-body=
Oct 30 01:28:17.313359 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:17.313186    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m" podUID="d77598a0-d6b3-421a-bd3d-5b13591ad686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused"
Oct 30 01:28:18.312672 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:18.312635    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" start-of-body=
Oct 30 01:28:18.312888 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:18.312690    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m" podUID="d77598a0-d6b3-421a-bd3d-5b13591ad686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused"
Oct 30 01:28:19.312955 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:19.312920    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" start-of-body=
Oct 30 01:28:19.313375 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:19.312973    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m" podUID="d77598a0-d6b3-421a-bd3d-5b13591ad686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused"
Oct 30 01:28:20.312670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:20.312632    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" start-of-body=
Oct 30 01:28:20.312872 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:20.312682    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m" podUID="d77598a0-d6b3-421a-bd3d-5b13591ad686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused"
Oct 30 01:28:21.795403 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:21.795373    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:21.795880 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:21.795829    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:21.796664 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:21.796644    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m"
Oct 30 01:28:22.138846 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.138769    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx"]
Oct 30 01:28:22.139260 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.139247    6099 cpu_manager.go:414] "containerMap: removing container" podUID="98f86982-37a2-45e5-8028-2bbc099ca613" containerName="revoke-keys"
Oct 30 01:28:22.139312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.139262    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98f86982-37a2-45e5-8028-2bbc099ca613" containerName="revoke-keys"
Oct 30 01:28:22.139350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.139329    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="98f86982-37a2-45e5-8028-2bbc099ca613" containerName="revoke-keys"
Oct 30 01:28:22.143880 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.143858    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx"
Oct 30 01:28:22.160076 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.160049    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:28:22.160315 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.160295    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:28:22.162834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.162815    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx"]
Oct 30 01:28:22.179127 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.179101    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc13f7e2-4f96-4993-8186-a287f49c4886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-x9phx\" (UID: \"dc13f7e2-4f96-4993-8186-a287f49c4886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx"
Oct 30 01:28:22.179249 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.179142    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgd87\" (UniqueName: \"kubernetes.io/projected/dc13f7e2-4f96-4993-8186-a287f49c4886-kube-api-access-vgd87\") pod \"maas-api-revoke-sub-ec843018-x9phx\" (UID: \"dc13f7e2-4f96-4993-8186-a287f49c4886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx"
Oct 30 01:28:22.179342 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.179328    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc13f7e2-4f96-4993-8186-a287f49c4886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-x9phx\" (UID: \"dc13f7e2-4f96-4993-8186-a287f49c4886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx"
Oct 30 01:28:22.186787 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.186767    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:28:22.186914 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.186852    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:28:22.280171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.280134    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc13f7e2-4f96-4993-8186-a287f49c4886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-x9phx\" (UID: \"dc13f7e2-4f96-4993-8186-a287f49c4886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx"
Oct 30 01:28:22.280355 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.280224    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc13f7e2-4f96-4993-8186-a287f49c4886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-x9phx\" (UID: \"dc13f7e2-4f96-4993-8186-a287f49c4886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx"
Oct 30 01:28:22.280355 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.280254    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgd87\" (UniqueName: \"kubernetes.io/projected/dc13f7e2-4f96-4993-8186-a287f49c4886-kube-api-access-vgd87\") pod \"maas-api-revoke-sub-ec843018-x9phx\" (UID: \"dc13f7e2-4f96-4993-8186-a287f49c4886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx"
Oct 30 01:28:22.280895 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.280875    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc13f7e2-4f96-4993-8186-a287f49c4886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-x9phx\" (UID: \"dc13f7e2-4f96-4993-8186-a287f49c4886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx"
Oct 30 01:28:22.282568 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.282539    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc13f7e2-4f96-4993-8186-a287f49c4886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-x9phx\" (UID: \"dc13f7e2-4f96-4993-8186-a287f49c4886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx"
Oct 30 01:28:22.312150 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.303623    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgd87\" (UniqueName: \"kubernetes.io/projected/dc13f7e2-4f96-4993-8186-a287f49c4886-kube-api-access-vgd87\") pod \"maas-api-revoke-sub-ec843018-x9phx\" (UID: \"dc13f7e2-4f96-4993-8186-a287f49c4886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx"
Oct 30 01:28:22.457851 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.457785    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx"
Oct 30 01:28:22.753920 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:22.753888    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx"]
Oct 30 01:28:22.754908 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:28:22.754881    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc13f7e2_4f96_4993_8186_a287f49c4886.slice/crio-8cdaab33297344b69a35499a7a006c0fc7e908b3ee9395d65c09ebda3829ed98 WatchSource:0}: Error finding container 8cdaab33297344b69a35499a7a006c0fc7e908b3ee9395d65c09ebda3829ed98: Status 404 returned error can't find the container with id 8cdaab33297344b69a35499a7a006c0fc7e908b3ee9395d65c09ebda3829ed98
Oct 30 01:28:23.545838 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:23.545797    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx" event={"ID":"dc13f7e2-4f96-4993-8186-a287f49c4886","Type":"ContainerStarted","Data":"1d168228f4e55158b7d60af8c9a3a041ec33a3902e4b8f5b380cc109a8238576"}
Oct 30 01:28:23.546315 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:23.545866    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx" event={"ID":"dc13f7e2-4f96-4993-8186-a287f49c4886","Type":"ContainerStarted","Data":"8cdaab33297344b69a35499a7a006c0fc7e908b3ee9395d65c09ebda3829ed98"}
Oct 30 01:28:23.586043 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:23.585996    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx" podStartSLOduration=1.585982895 podStartE2EDuration="1.585982895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:23.584510691 +0000 UTC m=+3197.441418449" watchObservedRunningTime="2026-10-30 01:28:23.585982895 +0000 UTC m=+3197.442890647"
Oct 30 01:28:23.814503 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:23.814423    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:28:24.553281 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:24.553248    6099 generic.go:356] "Generic (PLEG): container finished" podID="dc13f7e2-4f96-4993-8186-a287f49c4886" containerID="1d168228f4e55158b7d60af8c9a3a041ec33a3902e4b8f5b380cc109a8238576" exitCode=0
Oct 30 01:28:24.553682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:24.553340    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx" event={"ID":"dc13f7e2-4f96-4993-8186-a287f49c4886","Type":"ContainerDied","Data":"1d168228f4e55158b7d60af8c9a3a041ec33a3902e4b8f5b380cc109a8238576"}
Oct 30 01:28:25.703848 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:25.703820    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx"
Oct 30 01:28:25.827027 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:25.826931    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc13f7e2-4f96-4993-8186-a287f49c4886-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc13f7e2-4f96-4993-8186-a287f49c4886-maas-api-openshift-ca\") pod \"dc13f7e2-4f96-4993-8186-a287f49c4886\" (UID: \"dc13f7e2-4f96-4993-8186-a287f49c4886\") "
Oct 30 01:28:25.827027 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:25.826989    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc13f7e2-4f96-4993-8186-a287f49c4886-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc13f7e2-4f96-4993-8186-a287f49c4886-maas-api-xks-ca\") pod \"dc13f7e2-4f96-4993-8186-a287f49c4886\" (UID: \"dc13f7e2-4f96-4993-8186-a287f49c4886\") "
Oct 30 01:28:25.827027 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:25.827024    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc13f7e2-4f96-4993-8186-a287f49c4886-kube-api-access-vgd87\" (UniqueName: \"kubernetes.io/projected/dc13f7e2-4f96-4993-8186-a287f49c4886-kube-api-access-vgd87\") pod \"dc13f7e2-4f96-4993-8186-a287f49c4886\" (UID: \"dc13f7e2-4f96-4993-8186-a287f49c4886\") "
Oct 30 01:28:25.828100 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:25.827798    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc13f7e2-4f96-4993-8186-a287f49c4886-maas-api-openshift-ca" pod "dc13f7e2-4f96-4993-8186-a287f49c4886" (UID: "dc13f7e2-4f96-4993-8186-a287f49c4886"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:25.829994 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:25.829964    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc13f7e2-4f96-4993-8186-a287f49c4886-kube-api-access-vgd87" pod "dc13f7e2-4f96-4993-8186-a287f49c4886" (UID: "dc13f7e2-4f96-4993-8186-a287f49c4886"). InnerVolumeSpecName "kube-api-access-vgd87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:25.830502 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:25.830478    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc13f7e2-4f96-4993-8186-a287f49c4886-maas-api-xks-ca" pod "dc13f7e2-4f96-4993-8186-a287f49c4886" (UID: "dc13f7e2-4f96-4993-8186-a287f49c4886"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:28:25.928654 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:25.928611    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc13f7e2-4f96-4993-8186-a287f49c4886-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:28:25.928654 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:25.928654    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc13f7e2-4f96-4993-8186-a287f49c4886-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:28:25.928944 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:25.928674    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgd87\" (UniqueName: \"kubernetes.io/projected/dc13f7e2-4f96-4993-8186-a287f49c4886-kube-api-access-vgd87\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:28:26.564905 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:26.564877    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx"
Oct 30 01:28:26.565060 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:26.564910    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx" event={"ID":"dc13f7e2-4f96-4993-8186-a287f49c4886","Type":"ContainerDied","Data":"8cdaab33297344b69a35499a7a006c0fc7e908b3ee9395d65c09ebda3829ed98"}
Oct 30 01:28:26.565060 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:26.564955    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cdaab33297344b69a35499a7a006c0fc7e908b3ee9395d65c09ebda3829ed98"
Oct 30 01:28:28.038127 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:28.038093    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f"]
Oct 30 01:28:28.049758 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:28.049724    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a32a218-ktz5f"]
Oct 30 01:28:28.259898 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:28.259868    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cf5c354-53f3-4169-8c2b-42ea97f1b2a6" path="/var/lib/kubelet/pods/4cf5c354-53f3-4169-8c2b-42ea97f1b2a6/volumes"
Oct 30 01:28:37.713919 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:37.713881    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:28:46.632828 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:28:46.632785    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:28:50.111366 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.111335    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw"]
Oct 30 01:28:50.111834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.111818    6099 cpu_manager.go:414] "containerMap: removing container" podUID="dc13f7e2-4f96-4993-8186-a287f49c4886" containerName="revoke-keys"
Oct 30 01:28:50.111900 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.111835    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc13f7e2-4f96-4993-8186-a287f49c4886" containerName="revoke-keys"
Oct 30 01:28:50.111947 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.111935    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc13f7e2-4f96-4993-8186-a287f49c4886" containerName="revoke-keys"
Oct 30 01:28:50.115306 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.115289    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw"
Oct 30 01:28:50.131921 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.131897    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:28:50.143952 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.143931    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:28:50.144143 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.143951    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:28:50.144273 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.144209    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:28:50.145899 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.145880    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw"]
Oct 30 01:28:50.213780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.213749    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv45w\" (UniqueName: \"kubernetes.io/projected/3ea2543f-eb17-4c3b-a555-c6d8f866834a-kube-api-access-bv45w\") pod \"maas-api-revoke-sub-046e1343-zjvbw\" (UID: \"3ea2543f-eb17-4c3b-a555-c6d8f866834a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw"
Oct 30 01:28:50.213941 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.213798    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea2543f-eb17-4c3b-a555-c6d8f866834a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zjvbw\" (UID: \"3ea2543f-eb17-4c3b-a555-c6d8f866834a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw"
Oct 30 01:28:50.213941 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.213823    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea2543f-eb17-4c3b-a555-c6d8f866834a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zjvbw\" (UID: \"3ea2543f-eb17-4c3b-a555-c6d8f866834a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw"
Oct 30 01:28:50.314875 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.314840    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv45w\" (UniqueName: \"kubernetes.io/projected/3ea2543f-eb17-4c3b-a555-c6d8f866834a-kube-api-access-bv45w\") pod \"maas-api-revoke-sub-046e1343-zjvbw\" (UID: \"3ea2543f-eb17-4c3b-a555-c6d8f866834a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw"
Oct 30 01:28:50.315045 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.314893    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea2543f-eb17-4c3b-a555-c6d8f866834a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zjvbw\" (UID: \"3ea2543f-eb17-4c3b-a555-c6d8f866834a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw"
Oct 30 01:28:50.315045 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.314920    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea2543f-eb17-4c3b-a555-c6d8f866834a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zjvbw\" (UID: \"3ea2543f-eb17-4c3b-a555-c6d8f866834a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw"
Oct 30 01:28:50.315687 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.315661    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea2543f-eb17-4c3b-a555-c6d8f866834a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zjvbw\" (UID: \"3ea2543f-eb17-4c3b-a555-c6d8f866834a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw"
Oct 30 01:28:50.317288 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.317269    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea2543f-eb17-4c3b-a555-c6d8f866834a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zjvbw\" (UID: \"3ea2543f-eb17-4c3b-a555-c6d8f866834a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw"
Oct 30 01:28:50.338370 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.338347    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv45w\" (UniqueName: \"kubernetes.io/projected/3ea2543f-eb17-4c3b-a555-c6d8f866834a-kube-api-access-bv45w\") pod \"maas-api-revoke-sub-046e1343-zjvbw\" (UID: \"3ea2543f-eb17-4c3b-a555-c6d8f866834a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw"
Oct 30 01:28:50.428125 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.428064    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw"
Oct 30 01:28:50.723571 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:50.723542    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw"]
Oct 30 01:28:50.724401 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:28:50.724368    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea2543f_eb17_4c3b_a555_c6d8f866834a.slice/crio-f10e932a207dda70d4b2fbf191477a4b703ff59b4592ad1d4f606c22594bfb65 WatchSource:0}: Error finding container f10e932a207dda70d4b2fbf191477a4b703ff59b4592ad1d4f606c22594bfb65: Status 404 returned error can't find the container with id f10e932a207dda70d4b2fbf191477a4b703ff59b4592ad1d4f606c22594bfb65
Oct 30 01:28:51.714856 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:51.714759    6099 generic.go:356] "Generic (PLEG): container finished" podID="3ea2543f-eb17-4c3b-a555-c6d8f866834a" containerID="67a9a16419ef5876e08c90ae3cd70fc6204c8d4b8d27596462dcf6270d0054ac" exitCode=0
Oct 30 01:28:51.715210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:51.714844    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw" event={"ID":"3ea2543f-eb17-4c3b-a555-c6d8f866834a","Type":"ContainerDied","Data":"67a9a16419ef5876e08c90ae3cd70fc6204c8d4b8d27596462dcf6270d0054ac"}
Oct 30 01:28:51.715210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:51.714901    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw" event={"ID":"3ea2543f-eb17-4c3b-a555-c6d8f866834a","Type":"ContainerStarted","Data":"f10e932a207dda70d4b2fbf191477a4b703ff59b4592ad1d4f606c22594bfb65"}
Oct 30 01:28:52.856800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:52.856772    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw"
Oct 30 01:28:52.940764 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:52.940738    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea2543f-eb17-4c3b-a555-c6d8f866834a-kube-api-access-bv45w\" (UniqueName: \"kubernetes.io/projected/3ea2543f-eb17-4c3b-a555-c6d8f866834a-kube-api-access-bv45w\") pod \"3ea2543f-eb17-4c3b-a555-c6d8f866834a\" (UID: \"3ea2543f-eb17-4c3b-a555-c6d8f866834a\") "
Oct 30 01:28:52.940925 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:52.940805    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea2543f-eb17-4c3b-a555-c6d8f866834a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea2543f-eb17-4c3b-a555-c6d8f866834a-maas-api-openshift-ca\") pod \"3ea2543f-eb17-4c3b-a555-c6d8f866834a\" (UID: \"3ea2543f-eb17-4c3b-a555-c6d8f866834a\") "
Oct 30 01:28:52.940925 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:52.940838    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ea2543f-eb17-4c3b-a555-c6d8f866834a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea2543f-eb17-4c3b-a555-c6d8f866834a-maas-api-xks-ca\") pod \"3ea2543f-eb17-4c3b-a555-c6d8f866834a\" (UID: \"3ea2543f-eb17-4c3b-a555-c6d8f866834a\") "
Oct 30 01:28:52.941212 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:52.941184    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea2543f-eb17-4c3b-a555-c6d8f866834a-maas-api-openshift-ca" pod "3ea2543f-eb17-4c3b-a555-c6d8f866834a" (UID: "3ea2543f-eb17-4c3b-a555-c6d8f866834a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:52.941347 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:52.941292    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea2543f-eb17-4c3b-a555-c6d8f866834a-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:28:52.942763 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:52.942678    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea2543f-eb17-4c3b-a555-c6d8f866834a-kube-api-access-bv45w" pod "3ea2543f-eb17-4c3b-a555-c6d8f866834a" (UID: "3ea2543f-eb17-4c3b-a555-c6d8f866834a"). InnerVolumeSpecName "kube-api-access-bv45w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:52.943187 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:52.943161    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea2543f-eb17-4c3b-a555-c6d8f866834a-maas-api-xks-ca" pod "3ea2543f-eb17-4c3b-a555-c6d8f866834a" (UID: "3ea2543f-eb17-4c3b-a555-c6d8f866834a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:28:53.042622 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:53.042595    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv45w\" (UniqueName: \"kubernetes.io/projected/3ea2543f-eb17-4c3b-a555-c6d8f866834a-kube-api-access-bv45w\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:28:53.042622 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:53.042620    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea2543f-eb17-4c3b-a555-c6d8f866834a-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:28:53.727331 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:53.727302    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw"
Oct 30 01:28:53.727514 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:53.727339    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw" event={"ID":"3ea2543f-eb17-4c3b-a555-c6d8f866834a","Type":"ContainerDied","Data":"f10e932a207dda70d4b2fbf191477a4b703ff59b4592ad1d4f606c22594bfb65"}
Oct 30 01:28:53.727514 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:53.727387    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f10e932a207dda70d4b2fbf191477a4b703ff59b4592ad1d4f606c22594bfb65"
Oct 30 01:28:55.027803 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:55.027771    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n"]
Oct 30 01:28:55.039574 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:55.039539    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc666335-5fq9n"]
Oct 30 01:28:56.259833 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:28:56.259793    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cacfc24-be5d-4505-a761-18aa477444fb" path="/var/lib/kubelet/pods/4cacfc24-be5d-4505-a761-18aa477444fb/volumes"
Oct 30 01:29:00.416752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.416717    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg"]
Oct 30 01:29:00.417165 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.417140    6099 cpu_manager.go:414] "containerMap: removing container" podUID="3ea2543f-eb17-4c3b-a555-c6d8f866834a" containerName="revoke-keys"
Oct 30 01:29:00.417165 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.417152    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea2543f-eb17-4c3b-a555-c6d8f866834a" containerName="revoke-keys"
Oct 30 01:29:00.417258 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.417247    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea2543f-eb17-4c3b-a555-c6d8f866834a" containerName="revoke-keys"
Oct 30 01:29:00.421493 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.421474    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg"
Oct 30 01:29:00.449731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.449679    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:29:00.449874 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.449776    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:29:00.451500 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.451472    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:29:00.451624 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.451603    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:29:00.451731 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.451699    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg"]
Oct 30 01:29:00.513412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.513384    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f7f028-efa6-46f1-a697-ae84ea79b026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-7v9fg\" (UID: \"f3f7f028-efa6-46f1-a697-ae84ea79b026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg"
Oct 30 01:29:00.513537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.513426    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f7f028-efa6-46f1-a697-ae84ea79b026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-7v9fg\" (UID: \"f3f7f028-efa6-46f1-a697-ae84ea79b026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg"
Oct 30 01:29:00.513537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.513474    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vpjj\" (UniqueName: \"kubernetes.io/projected/f3f7f028-efa6-46f1-a697-ae84ea79b026-kube-api-access-6vpjj\") pod \"maas-api-revoke-sub-d3fba056-7v9fg\" (UID: \"f3f7f028-efa6-46f1-a697-ae84ea79b026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg"
Oct 30 01:29:00.614621 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.614591    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f7f028-efa6-46f1-a697-ae84ea79b026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-7v9fg\" (UID: \"f3f7f028-efa6-46f1-a697-ae84ea79b026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg"
Oct 30 01:29:00.614769 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.614631    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f7f028-efa6-46f1-a697-ae84ea79b026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-7v9fg\" (UID: \"f3f7f028-efa6-46f1-a697-ae84ea79b026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg"
Oct 30 01:29:00.614769 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.614661    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vpjj\" (UniqueName: \"kubernetes.io/projected/f3f7f028-efa6-46f1-a697-ae84ea79b026-kube-api-access-6vpjj\") pod \"maas-api-revoke-sub-d3fba056-7v9fg\" (UID: \"f3f7f028-efa6-46f1-a697-ae84ea79b026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg"
Oct 30 01:29:00.615371 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.615352    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f7f028-efa6-46f1-a697-ae84ea79b026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-7v9fg\" (UID: \"f3f7f028-efa6-46f1-a697-ae84ea79b026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg"
Oct 30 01:29:00.617083 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.617064    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f7f028-efa6-46f1-a697-ae84ea79b026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-7v9fg\" (UID: \"f3f7f028-efa6-46f1-a697-ae84ea79b026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg"
Oct 30 01:29:00.638167 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.638141    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vpjj\" (UniqueName: \"kubernetes.io/projected/f3f7f028-efa6-46f1-a697-ae84ea79b026-kube-api-access-6vpjj\") pod \"maas-api-revoke-sub-d3fba056-7v9fg\" (UID: \"f3f7f028-efa6-46f1-a697-ae84ea79b026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg"
Oct 30 01:29:00.734591 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:00.734532    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg"
Oct 30 01:29:01.032379 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:01.032348    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg"]
Oct 30 01:29:01.033888 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:29:01.033860    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f7f028_efa6_46f1_a697_ae84ea79b026.slice/crio-fc6ac0f62fbfd04c247e830e7284d72720830b49101d3876c0f44613b45b4fbd WatchSource:0}: Error finding container fc6ac0f62fbfd04c247e830e7284d72720830b49101d3876c0f44613b45b4fbd: Status 404 returned error can't find the container with id fc6ac0f62fbfd04c247e830e7284d72720830b49101d3876c0f44613b45b4fbd
Oct 30 01:29:01.331638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:01.331607    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:29:01.779436 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:01.779334    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg" event={"ID":"f3f7f028-efa6-46f1-a697-ae84ea79b026","Type":"ContainerStarted","Data":"ddf61b3f80f951b7c76b89f5b910bb26d5a71974e9b30b67cfd04eaa748ebfe2"}
Oct 30 01:29:01.779436 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:01.779399    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg" event={"ID":"f3f7f028-efa6-46f1-a697-ae84ea79b026","Type":"ContainerStarted","Data":"fc6ac0f62fbfd04c247e830e7284d72720830b49101d3876c0f44613b45b4fbd"}
Oct 30 01:29:01.819894 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:01.819824    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg" podStartSLOduration=1.8198070720000001 podStartE2EDuration="1.819807072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:01.818783446 +0000 UTC m=+3235.675691200" watchObservedRunningTime="2026-10-30 01:29:01.819807072 +0000 UTC m=+3235.676714828"
Oct 30 01:29:02.813730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:02.813114    6099 generic.go:356] "Generic (PLEG): container finished" podID="f3f7f028-efa6-46f1-a697-ae84ea79b026" containerID="ddf61b3f80f951b7c76b89f5b910bb26d5a71974e9b30b67cfd04eaa748ebfe2" exitCode=0
Oct 30 01:29:02.813730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:02.813287    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg" event={"ID":"f3f7f028-efa6-46f1-a697-ae84ea79b026","Type":"ContainerDied","Data":"ddf61b3f80f951b7c76b89f5b910bb26d5a71974e9b30b67cfd04eaa748ebfe2"}
Oct 30 01:29:04.256933 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:04.256900    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg"
Oct 30 01:29:04.361939 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:04.361908    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3f7f028-efa6-46f1-a697-ae84ea79b026-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f7f028-efa6-46f1-a697-ae84ea79b026-maas-api-xks-ca\") pod \"f3f7f028-efa6-46f1-a697-ae84ea79b026\" (UID: \"f3f7f028-efa6-46f1-a697-ae84ea79b026\") "
Oct 30 01:29:04.362125 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:04.362013    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f7f028-efa6-46f1-a697-ae84ea79b026-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f7f028-efa6-46f1-a697-ae84ea79b026-maas-api-openshift-ca\") pod \"f3f7f028-efa6-46f1-a697-ae84ea79b026\" (UID: \"f3f7f028-efa6-46f1-a697-ae84ea79b026\") "
Oct 30 01:29:04.362125 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:04.362035    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f7f028-efa6-46f1-a697-ae84ea79b026-kube-api-access-6vpjj\" (UniqueName: \"kubernetes.io/projected/f3f7f028-efa6-46f1-a697-ae84ea79b026-kube-api-access-6vpjj\") pod \"f3f7f028-efa6-46f1-a697-ae84ea79b026\" (UID: \"f3f7f028-efa6-46f1-a697-ae84ea79b026\") "
Oct 30 01:29:04.362443 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:04.362410    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f7f028-efa6-46f1-a697-ae84ea79b026-maas-api-openshift-ca" pod "f3f7f028-efa6-46f1-a697-ae84ea79b026" (UID: "f3f7f028-efa6-46f1-a697-ae84ea79b026"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:04.364121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:04.364098    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f7f028-efa6-46f1-a697-ae84ea79b026-maas-api-xks-ca" pod "f3f7f028-efa6-46f1-a697-ae84ea79b026" (UID: "f3f7f028-efa6-46f1-a697-ae84ea79b026"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:29:04.364287 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:04.364225    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f7f028-efa6-46f1-a697-ae84ea79b026-kube-api-access-6vpjj" pod "f3f7f028-efa6-46f1-a697-ae84ea79b026" (UID: "f3f7f028-efa6-46f1-a697-ae84ea79b026"). InnerVolumeSpecName "kube-api-access-6vpjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:04.463753 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:04.463685    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f7f028-efa6-46f1-a697-ae84ea79b026-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.463753 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:04.463732    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f7f028-efa6-46f1-a697-ae84ea79b026-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.463753 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:04.463742    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vpjj\" (UniqueName: \"kubernetes.io/projected/f3f7f028-efa6-46f1-a697-ae84ea79b026-kube-api-access-6vpjj\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.851784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:04.851754    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg"
Oct 30 01:29:04.851968 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:04.851755    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg" event={"ID":"f3f7f028-efa6-46f1-a697-ae84ea79b026","Type":"ContainerDied","Data":"fc6ac0f62fbfd04c247e830e7284d72720830b49101d3876c0f44613b45b4fbd"}
Oct 30 01:29:04.851968 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:04.851859    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc6ac0f62fbfd04c247e830e7284d72720830b49101d3876c0f44613b45b4fbd"
Oct 30 01:29:11.658879 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:11.658840    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:29:15.510202 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:15.510171    6099 scope.go:122] "RemoveContainer" containerID="8143995498b045c1d9897d7ec39c81537ec2e0bcb1a1f2fb10c014d6162ffe07"
Oct 30 01:29:15.519319 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:15.519293    6099 scope.go:122] "RemoveContainer" containerID="a25added2664f82202c6b91648014662deaaf45644976fa9e484cfa3b70ef2b3"
Oct 30 01:29:15.527458 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:15.527438    6099 scope.go:122] "RemoveContainer" containerID="48a909c8ae54b945ddb3b66815cf81bb0673316162271f395d454cbee780038b"
Oct 30 01:29:15.535190 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:15.535166    6099 scope.go:122] "RemoveContainer" containerID="c7d2cc1e91152395fcfade760fe19629ac9a4c103d99d13c9c441247bcfefea8"
Oct 30 01:29:15.542869 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:15.542853    6099 scope.go:122] "RemoveContainer" containerID="d188440e0abafffa1bd8dbf43a678fed7c7aa3b9703580bfbf94b002bc4ed4a8"
Oct 30 01:29:15.550562 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:15.550546    6099 scope.go:122] "RemoveContainer" containerID="ea4e05531671949b56d171ff90180c41a9e221e648926d3df3fc6b8f02ef4d03"
Oct 30 01:29:16.044301 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:16.044269    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:29:16.634552 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:29:16.634511    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:29:21.734361 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.734273    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6"]
Oct 30 01:29:21.734875 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.734763    6099 cpu_manager.go:414] "containerMap: removing container" podUID="f3f7f028-efa6-46f1-a697-ae84ea79b026" containerName="revoke-keys"
Oct 30 01:29:21.734875 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.734775    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f7f028-efa6-46f1-a697-ae84ea79b026" containerName="revoke-keys"
Oct 30 01:29:21.734875 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.734862    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f7f028-efa6-46f1-a697-ae84ea79b026" containerName="revoke-keys"
Oct 30 01:29:21.740240 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.740219    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6"
Oct 30 01:29:21.755522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.755466    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:29:21.755522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.755468    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:29:21.767457 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.767427    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:29:21.767586 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.767476    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:29:21.769784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.769758    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6"]
Oct 30 01:29:21.869793 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.869758    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-4klp6\" (UID: \"cdd89c92-dad3-4b87-9a44-7843ecd7b38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6"
Oct 30 01:29:21.869793 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.869793    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-4klp6\" (UID: \"cdd89c92-dad3-4b87-9a44-7843ecd7b38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6"
Oct 30 01:29:21.870066 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.869995    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzkdn\" (UniqueName: \"kubernetes.io/projected/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-kube-api-access-dzkdn\") pod \"maas-api-revoke-sub-ae594979-4klp6\" (UID: \"cdd89c92-dad3-4b87-9a44-7843ecd7b38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6"
Oct 30 01:29:21.971576 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.971550    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-4klp6\" (UID: \"cdd89c92-dad3-4b87-9a44-7843ecd7b38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6"
Oct 30 01:29:21.971760 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.971581    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-4klp6\" (UID: \"cdd89c92-dad3-4b87-9a44-7843ecd7b38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6"
Oct 30 01:29:21.971760 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.971664    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkdn\" (UniqueName: \"kubernetes.io/projected/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-kube-api-access-dzkdn\") pod \"maas-api-revoke-sub-ae594979-4klp6\" (UID: \"cdd89c92-dad3-4b87-9a44-7843ecd7b38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6"
Oct 30 01:29:21.972281 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.972253    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-4klp6\" (UID: \"cdd89c92-dad3-4b87-9a44-7843ecd7b38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6"
Oct 30 01:29:21.973911 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.973887    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-4klp6\" (UID: \"cdd89c92-dad3-4b87-9a44-7843ecd7b38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6"
Oct 30 01:29:21.994685 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:21.994633    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzkdn\" (UniqueName: \"kubernetes.io/projected/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-kube-api-access-dzkdn\") pod \"maas-api-revoke-sub-ae594979-4klp6\" (UID: \"cdd89c92-dad3-4b87-9a44-7843ecd7b38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6"
Oct 30 01:29:22.054121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:22.054098    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6"
Oct 30 01:29:22.354804 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:22.354772    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6"]
Oct 30 01:29:22.978740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:22.978633    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6" event={"ID":"cdd89c92-dad3-4b87-9a44-7843ecd7b38c","Type":"ContainerStarted","Data":"723eb9d772e36b5f3b336e1f5dbe63ec21031b11f12f4b78a824e765b5121f7b"}
Oct 30 01:29:22.978740 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:22.978695    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6" event={"ID":"cdd89c92-dad3-4b87-9a44-7843ecd7b38c","Type":"ContainerStarted","Data":"83f77c3e028f2991ea0cbac2effeec6fc3e9c3a4aa380e463a1aa36517350a1c"}
Oct 30 01:29:23.020348 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:23.020292    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6" podStartSLOduration=2.020279057 podStartE2EDuration="2.020279057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:23.018441324 +0000 UTC m=+3256.875349080" watchObservedRunningTime="2026-10-30 01:29:23.020279057 +0000 UTC m=+3256.877186845"
Oct 30 01:29:23.986155 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:23.986124    6099 generic.go:356] "Generic (PLEG): container finished" podID="cdd89c92-dad3-4b87-9a44-7843ecd7b38c" containerID="723eb9d772e36b5f3b336e1f5dbe63ec21031b11f12f4b78a824e765b5121f7b" exitCode=0
Oct 30 01:29:23.986547 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:23.986211    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6" event={"ID":"cdd89c92-dad3-4b87-9a44-7843ecd7b38c","Type":"ContainerDied","Data":"723eb9d772e36b5f3b336e1f5dbe63ec21031b11f12f4b78a824e765b5121f7b"}
Oct 30 01:29:25.130700 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:25.130673    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6"
Oct 30 01:29:25.303595 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:25.303567    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-maas-api-openshift-ca\") pod \"cdd89c92-dad3-4b87-9a44-7843ecd7b38c\" (UID: \"cdd89c92-dad3-4b87-9a44-7843ecd7b38c\") "
Oct 30 01:29:25.303770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:25.303722    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-maas-api-xks-ca\") pod \"cdd89c92-dad3-4b87-9a44-7843ecd7b38c\" (UID: \"cdd89c92-dad3-4b87-9a44-7843ecd7b38c\") "
Oct 30 01:29:25.303826 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:25.303774    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-kube-api-access-dzkdn\" (UniqueName: \"kubernetes.io/projected/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-kube-api-access-dzkdn\") pod \"cdd89c92-dad3-4b87-9a44-7843ecd7b38c\" (UID: \"cdd89c92-dad3-4b87-9a44-7843ecd7b38c\") "
Oct 30 01:29:25.304011 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:25.303985    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-maas-api-openshift-ca" pod "cdd89c92-dad3-4b87-9a44-7843ecd7b38c" (UID: "cdd89c92-dad3-4b87-9a44-7843ecd7b38c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:25.304165 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:25.304148    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:29:25.305802 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:25.305775    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-maas-api-xks-ca" pod "cdd89c92-dad3-4b87-9a44-7843ecd7b38c" (UID: "cdd89c92-dad3-4b87-9a44-7843ecd7b38c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:29:25.305908 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:25.305871    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-kube-api-access-dzkdn" pod "cdd89c92-dad3-4b87-9a44-7843ecd7b38c" (UID: "cdd89c92-dad3-4b87-9a44-7843ecd7b38c"). InnerVolumeSpecName "kube-api-access-dzkdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:25.405377 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:25.405344    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:29:25.405377 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:25.405372    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzkdn\" (UniqueName: \"kubernetes.io/projected/cdd89c92-dad3-4b87-9a44-7843ecd7b38c-kube-api-access-dzkdn\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:29:25.998209 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:25.998171    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6" event={"ID":"cdd89c92-dad3-4b87-9a44-7843ecd7b38c","Type":"ContainerDied","Data":"83f77c3e028f2991ea0cbac2effeec6fc3e9c3a4aa380e463a1aa36517350a1c"}
Oct 30 01:29:25.998209 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:25.998191    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6"
Oct 30 01:29:25.998209 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:25.998218    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83f77c3e028f2991ea0cbac2effeec6fc3e9c3a4aa380e463a1aa36517350a1c"
Oct 30 01:29:32.937513 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:32.937482    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:29:34.033423 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:34.033390    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x"]
Oct 30 01:29:34.046279 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:34.046251    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13935af2-npw5x"]
Oct 30 01:29:34.260489 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:34.260457    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f597c60b-9751-46a4-b4da-7b82c6fa4b1e" path="/var/lib/kubelet/pods/f597c60b-9751-46a4-b4da-7b82c6fa4b1e/volumes"
Oct 30 01:29:37.841830 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:37.841790    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:29:37.842476 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:37.842443    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-pc67m" podUID="d77598a0-d6b3-421a-bd3d-5b13591ad686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:29:38.007970 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:38.007925    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:29:43.070025 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.069983    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td"]
Oct 30 01:29:43.070537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.070462    6099 cpu_manager.go:414] "containerMap: removing container" podUID="cdd89c92-dad3-4b87-9a44-7843ecd7b38c" containerName="revoke-keys"
Oct 30 01:29:43.070537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.070476    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd89c92-dad3-4b87-9a44-7843ecd7b38c" containerName="revoke-keys"
Oct 30 01:29:43.070617 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.070553    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd89c92-dad3-4b87-9a44-7843ecd7b38c" containerName="revoke-keys"
Oct 30 01:29:43.073895 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.073878    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td"
Oct 30 01:29:43.091078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.091032    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:29:43.096620 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.096591    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9eec167-9b07-42d0-b70d-658e5b006009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-jm5td\" (UID: \"a9eec167-9b07-42d0-b70d-658e5b006009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td"
Oct 30 01:29:43.096777 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.096749    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56m5t\" (UniqueName: \"kubernetes.io/projected/a9eec167-9b07-42d0-b70d-658e5b006009-kube-api-access-56m5t\") pod \"maas-api-revoke-sub-a4aef7d7-jm5td\" (UID: \"a9eec167-9b07-42d0-b70d-658e5b006009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td"
Oct 30 01:29:43.096841 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.096822    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9eec167-9b07-42d0-b70d-658e5b006009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-jm5td\" (UID: \"a9eec167-9b07-42d0-b70d-658e5b006009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td"
Oct 30 01:29:43.103056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.103015    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:29:43.103056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.103032    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:29:43.103056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.103014    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:29:43.105137 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.105112    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td"]
Oct 30 01:29:43.198060 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.198030    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9eec167-9b07-42d0-b70d-658e5b006009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-jm5td\" (UID: \"a9eec167-9b07-42d0-b70d-658e5b006009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td"
Oct 30 01:29:43.198239 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.198108    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56m5t\" (UniqueName: \"kubernetes.io/projected/a9eec167-9b07-42d0-b70d-658e5b006009-kube-api-access-56m5t\") pod \"maas-api-revoke-sub-a4aef7d7-jm5td\" (UID: \"a9eec167-9b07-42d0-b70d-658e5b006009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td"
Oct 30 01:29:43.198289 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.198256    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9eec167-9b07-42d0-b70d-658e5b006009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-jm5td\" (UID: \"a9eec167-9b07-42d0-b70d-658e5b006009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td"
Oct 30 01:29:43.198766 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.198740    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9eec167-9b07-42d0-b70d-658e5b006009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-jm5td\" (UID: \"a9eec167-9b07-42d0-b70d-658e5b006009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td"
Oct 30 01:29:43.200668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.200639    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9eec167-9b07-42d0-b70d-658e5b006009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-jm5td\" (UID: \"a9eec167-9b07-42d0-b70d-658e5b006009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td"
Oct 30 01:29:43.221305 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.221273    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56m5t\" (UniqueName: \"kubernetes.io/projected/a9eec167-9b07-42d0-b70d-658e5b006009-kube-api-access-56m5t\") pod \"maas-api-revoke-sub-a4aef7d7-jm5td\" (UID: \"a9eec167-9b07-42d0-b70d-658e5b006009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td"
Oct 30 01:29:43.386155 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.386077    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td"
Oct 30 01:29:43.685682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:43.685649    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td"]
Oct 30 01:29:43.686813 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:29:43.686787    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9eec167_9b07_42d0_b70d_658e5b006009.slice/crio-3b56265384da95712a0c2662a6595c3dbdc14b6631e9617599de4b1c7ec04e17 WatchSource:0}: Error finding container 3b56265384da95712a0c2662a6595c3dbdc14b6631e9617599de4b1c7ec04e17: Status 404 returned error can't find the container with id 3b56265384da95712a0c2662a6595c3dbdc14b6631e9617599de4b1c7ec04e17
Oct 30 01:29:44.152694 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:44.152658    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td" event={"ID":"a9eec167-9b07-42d0-b70d-658e5b006009","Type":"ContainerStarted","Data":"2e55d6fe9adf6c6aa881790e41af17ddc943ee0d6d4a31a8e13f360b93fded6b"}
Oct 30 01:29:44.153108 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:44.152731    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td" event={"ID":"a9eec167-9b07-42d0-b70d-658e5b006009","Type":"ContainerStarted","Data":"3b56265384da95712a0c2662a6595c3dbdc14b6631e9617599de4b1c7ec04e17"}
Oct 30 01:29:44.194943 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:44.194895    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td" podStartSLOduration=1.194882547 podStartE2EDuration="1.194882547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:44.193459269 +0000 UTC m=+3278.050367024" watchObservedRunningTime="2026-10-30 01:29:44.194882547 +0000 UTC m=+3278.051790301"
Oct 30 01:29:45.160107 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:45.160070    6099 generic.go:356] "Generic (PLEG): container finished" podID="a9eec167-9b07-42d0-b70d-658e5b006009" containerID="2e55d6fe9adf6c6aa881790e41af17ddc943ee0d6d4a31a8e13f360b93fded6b" exitCode=0
Oct 30 01:29:45.160526 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:45.160157    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td" event={"ID":"a9eec167-9b07-42d0-b70d-658e5b006009","Type":"ContainerDied","Data":"2e55d6fe9adf6c6aa881790e41af17ddc943ee0d6d4a31a8e13f360b93fded6b"}
Oct 30 01:29:46.300080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:46.300056    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td"
Oct 30 01:29:46.328855 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:46.328828    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9eec167-9b07-42d0-b70d-658e5b006009-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9eec167-9b07-42d0-b70d-658e5b006009-maas-api-xks-ca\") pod \"a9eec167-9b07-42d0-b70d-658e5b006009\" (UID: \"a9eec167-9b07-42d0-b70d-658e5b006009\") "
Oct 30 01:29:46.329062 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:46.329040    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9eec167-9b07-42d0-b70d-658e5b006009-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9eec167-9b07-42d0-b70d-658e5b006009-maas-api-openshift-ca\") pod \"a9eec167-9b07-42d0-b70d-658e5b006009\" (UID: \"a9eec167-9b07-42d0-b70d-658e5b006009\") "
Oct 30 01:29:46.329147 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:46.329075    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9eec167-9b07-42d0-b70d-658e5b006009-kube-api-access-56m5t\" (UniqueName: \"kubernetes.io/projected/a9eec167-9b07-42d0-b70d-658e5b006009-kube-api-access-56m5t\") pod \"a9eec167-9b07-42d0-b70d-658e5b006009\" (UID: \"a9eec167-9b07-42d0-b70d-658e5b006009\") "
Oct 30 01:29:46.329415 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:46.329395    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9eec167-9b07-42d0-b70d-658e5b006009-maas-api-openshift-ca" pod "a9eec167-9b07-42d0-b70d-658e5b006009" (UID: "a9eec167-9b07-42d0-b70d-658e5b006009"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:46.330987 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:46.330960    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9eec167-9b07-42d0-b70d-658e5b006009-maas-api-xks-ca" pod "a9eec167-9b07-42d0-b70d-658e5b006009" (UID: "a9eec167-9b07-42d0-b70d-658e5b006009"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:29:46.331078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:46.331059    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9eec167-9b07-42d0-b70d-658e5b006009-kube-api-access-56m5t" pod "a9eec167-9b07-42d0-b70d-658e5b006009" (UID: "a9eec167-9b07-42d0-b70d-658e5b006009"). InnerVolumeSpecName "kube-api-access-56m5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:46.430218 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:46.430125    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56m5t\" (UniqueName: \"kubernetes.io/projected/a9eec167-9b07-42d0-b70d-658e5b006009-kube-api-access-56m5t\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:29:46.430218 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:46.430157    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9eec167-9b07-42d0-b70d-658e5b006009-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:29:46.430218 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:46.430170    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9eec167-9b07-42d0-b70d-658e5b006009-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:29:46.629322 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:29:46.629286    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:29:47.172816 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:47.172782    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td"
Oct 30 01:29:47.173018 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:47.172810    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td" event={"ID":"a9eec167-9b07-42d0-b70d-658e5b006009","Type":"ContainerDied","Data":"3b56265384da95712a0c2662a6595c3dbdc14b6631e9617599de4b1c7ec04e17"}
Oct 30 01:29:47.173018 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:47.172874    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b56265384da95712a0c2662a6595c3dbdc14b6631e9617599de4b1c7ec04e17"
Oct 30 01:29:54.252614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:54.252577    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:29:58.537412 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:29:58.537378    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:30:00.145132 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:00.145090    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq"]
Oct 30 01:30:00.145649 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:00.145542    6099 cpu_manager.go:414] "containerMap: removing container" podUID="a9eec167-9b07-42d0-b70d-658e5b006009" containerName="revoke-keys"
Oct 30 01:30:00.145649 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:00.145554    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9eec167-9b07-42d0-b70d-658e5b006009" containerName="revoke-keys"
Oct 30 01:30:00.145649 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:00.145644    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9eec167-9b07-42d0-b70d-658e5b006009" containerName="revoke-keys"
Oct 30 01:30:00.149505 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:00.149479    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq"
Oct 30 01:30:00.183738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:00.175570    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:30:00.183738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:00.175667    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:30:00.183738 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:00.175838    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:30:00.184691 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:00.184664    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq"]
Oct 30 01:30:00.268730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:00.264915    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a467ea6-394a-4538-b186-efaf14574ccb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888730-jn7tq\" (UID: \"5a467ea6-394a-4538-b186-efaf14574ccb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq"
Oct 30 01:30:00.268730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:00.265123    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mffn\" (UniqueName: \"kubernetes.io/projected/5a467ea6-394a-4538-b186-efaf14574ccb-kube-api-access-6mffn\") pod \"maas-api-key-cleanup-29888730-jn7tq\" (UID: \"5a467ea6-394a-4538-b186-efaf14574ccb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq"
Oct 30 01:30:00.369307 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:00.369262    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mffn\" (UniqueName: \"kubernetes.io/projected/5a467ea6-394a-4538-b186-efaf14574ccb-kube-api-access-6mffn\") pod \"maas-api-key-cleanup-29888730-jn7tq\" (UID: \"5a467ea6-394a-4538-b186-efaf14574ccb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq"
Oct 30 01:30:00.370064 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:00.370034    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a467ea6-394a-4538-b186-efaf14574ccb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888730-jn7tq\" (UID: \"5a467ea6-394a-4538-b186-efaf14574ccb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq"
Oct 30 01:30:00.382964 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:00.371021    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a467ea6-394a-4538-b186-efaf14574ccb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888730-jn7tq\" (UID: \"5a467ea6-394a-4538-b186-efaf14574ccb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq"
Oct 30 01:30:00.427771 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:00.418447    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mffn\" (UniqueName: \"kubernetes.io/projected/5a467ea6-394a-4538-b186-efaf14574ccb-kube-api-access-6mffn\") pod \"maas-api-key-cleanup-29888730-jn7tq\" (UID: \"5a467ea6-394a-4538-b186-efaf14574ccb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq"
Oct 30 01:30:00.462425 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:00.462386    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq"
Oct 30 01:30:00.902729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:00.899138    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq"]
Oct 30 01:30:00.935364 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:30:00.935316    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a467ea6_394a_4538_b186_efaf14574ccb.slice/crio-86c437fbf3da54f34c0dbef8577024e011bbcbf412fb277bf36c6108fc4a3dc7 WatchSource:0}: Error finding container 86c437fbf3da54f34c0dbef8577024e011bbcbf412fb277bf36c6108fc4a3dc7: Status 404 returned error can't find the container with id 86c437fbf3da54f34c0dbef8577024e011bbcbf412fb277bf36c6108fc4a3dc7
Oct 30 01:30:01.149860 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:01.149820    6099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:30:01.150242 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:01.149870    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7nffr" podUID="1f88f534-38ba-41ff-b2e5-b7335f2f1fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:30:01.260123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:01.260091    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq" event={"ID":"5a467ea6-394a-4538-b186-efaf14574ccb","Type":"ContainerStarted","Data":"86c437fbf3da54f34c0dbef8577024e011bbcbf412fb277bf36c6108fc4a3dc7"}
Oct 30 01:30:02.267659 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:02.267571    6099 generic.go:356] "Generic (PLEG): container finished" podID="5a467ea6-394a-4538-b186-efaf14574ccb" containerID="e6c4403b008725ea036a1846352dfe910ff35486f8ba9e7243bc63215ecd30f9" exitCode=60
Oct 30 01:30:02.268163 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:02.267654    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq" event={"ID":"5a467ea6-394a-4538-b186-efaf14574ccb","Type":"ContainerDied","Data":"e6c4403b008725ea036a1846352dfe910ff35486f8ba9e7243bc63215ecd30f9"}
Oct 30 01:30:02.268163 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:02.267909    6099 scope.go:122] "RemoveContainer" containerID="e6c4403b008725ea036a1846352dfe910ff35486f8ba9e7243bc63215ecd30f9"
Oct 30 01:30:03.275682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:03.275601    6099 generic.go:356] "Generic (PLEG): container finished" podID="5a467ea6-394a-4538-b186-efaf14574ccb" containerID="64289f2e5317857aa23225001ec8da706f80aceff29fc63e05b2ecb7944aa89c" exitCode=60
Oct 30 01:30:03.275682 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:03.275647    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq" event={"ID":"5a467ea6-394a-4538-b186-efaf14574ccb","Type":"ContainerDied","Data":"64289f2e5317857aa23225001ec8da706f80aceff29fc63e05b2ecb7944aa89c"}
Oct 30 01:30:03.276150 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:03.275688    6099 scope.go:122] "RemoveContainer" containerID="e6c4403b008725ea036a1846352dfe910ff35486f8ba9e7243bc63215ecd30f9"
Oct 30 01:30:03.276150 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:03.275853    6099 scope.go:122] "RemoveContainer" containerID="64289f2e5317857aa23225001ec8da706f80aceff29fc63e05b2ecb7944aa89c"
Oct 30 01:30:03.276150 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:30:03.276074    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888730-jn7tq_odh-ai-gateway-infra(5a467ea6-394a-4538-b186-efaf14574ccb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq" podUID="5a467ea6-394a-4538-b186-efaf14574ccb"
Oct 30 01:30:04.282824 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:04.282790    6099 scope.go:122] "RemoveContainer" containerID="64289f2e5317857aa23225001ec8da706f80aceff29fc63e05b2ecb7944aa89c"
Oct 30 01:30:04.283230 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:30:04.283019    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888730-jn7tq_odh-ai-gateway-infra(5a467ea6-394a-4538-b186-efaf14574ccb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq" podUID="5a467ea6-394a-4538-b186-efaf14574ccb"
Oct 30 01:30:06.969741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:06.969693    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 01:30:06.984049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:06.984027    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 01:30:14.254241 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:14.254210    6099 scope.go:122] "RemoveContainer" containerID="64289f2e5317857aa23225001ec8da706f80aceff29fc63e05b2ecb7944aa89c"
Oct 30 01:30:15.117268 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:15.117232    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:30:15.344766 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:15.344682    6099 generic.go:356] "Generic (PLEG): container finished" podID="5a467ea6-394a-4538-b186-efaf14574ccb" containerID="662d4b9889b09bcb9a6fee92ccb4a923f3757b502b0cd133763a0e74d1b25735" exitCode=60
Oct 30 01:30:15.344766 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:15.344755    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq" event={"ID":"5a467ea6-394a-4538-b186-efaf14574ccb","Type":"ContainerDied","Data":"662d4b9889b09bcb9a6fee92ccb4a923f3757b502b0cd133763a0e74d1b25735"}
Oct 30 01:30:15.345231 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:15.344816    6099 scope.go:122] "RemoveContainer" containerID="64289f2e5317857aa23225001ec8da706f80aceff29fc63e05b2ecb7944aa89c"
Oct 30 01:30:15.345231 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:15.345140    6099 scope.go:122] "RemoveContainer" containerID="662d4b9889b09bcb9a6fee92ccb4a923f3757b502b0cd133763a0e74d1b25735"
Oct 30 01:30:15.345457 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:30:15.345432    6099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888730-jn7tq_odh-ai-gateway-infra(5a467ea6-394a-4538-b186-efaf14574ccb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq" podUID="5a467ea6-394a-4538-b186-efaf14574ccb"
Oct 30 01:30:15.590836 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:15.590799    6099 scope.go:122] "RemoveContainer" containerID="aa25e931f4ba953eea471689851f1973958350a700b83589953a9e00ac00b2b3"
Oct 30 01:30:15.602140 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:15.602111    6099 scope.go:122] "RemoveContainer" containerID="b41ba09aeb529035c0851101f185c098b93c07ef2d9784f7e87aabd3d4358101"
Oct 30 01:30:15.612700 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:15.612676    6099 scope.go:122] "RemoveContainer" containerID="7abbefdaafb1c394cb16963f5f0b9afe921185cc83a82daddb121978af1961c7"
Oct 30 01:30:15.623296 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:15.623260    6099 scope.go:122] "RemoveContainer" containerID="ad46b6823071dcc5745541ed04bae8afdf0e810025d418d201b66d38ffe7bc45"
Oct 30 01:30:15.634776 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:15.634750    6099 scope.go:122] "RemoveContainer" containerID="de4480663c8df7dd8d95d6999ed086c98d1e7bde23b340610c46082236beea1a"
Oct 30 01:30:15.645688 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:15.645661    6099 scope.go:122] "RemoveContainer" containerID="f41dbcb82599c842b56a04742d1afaf1ea674a9540016f22419d78183cc7e9be"
Oct 30 01:30:16.393910 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:16.393880    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq"]
Oct 30 01:30:16.677724 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:16.677414    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq"
Oct 30 01:30:16.732235 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:30:16.732195    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:30:16.817733 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:16.808888    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a467ea6-394a-4538-b186-efaf14574ccb-kube-api-access-6mffn\" (UniqueName: \"kubernetes.io/projected/5a467ea6-394a-4538-b186-efaf14574ccb-kube-api-access-6mffn\") pod \"5a467ea6-394a-4538-b186-efaf14574ccb\" (UID: \"5a467ea6-394a-4538-b186-efaf14574ccb\") "
Oct 30 01:30:16.841863 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:16.841829    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a467ea6-394a-4538-b186-efaf14574ccb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a467ea6-394a-4538-b186-efaf14574ccb-maas-api-service-ca\") pod \"5a467ea6-394a-4538-b186-efaf14574ccb\" (UID: \"5a467ea6-394a-4538-b186-efaf14574ccb\") "
Oct 30 01:30:16.842633 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:16.842602    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a467ea6-394a-4538-b186-efaf14574ccb-maas-api-service-ca" pod "5a467ea6-394a-4538-b186-efaf14574ccb" (UID: "5a467ea6-394a-4538-b186-efaf14574ccb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:16.846496 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:16.846221    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a467ea6-394a-4538-b186-efaf14574ccb-kube-api-access-6mffn" pod "5a467ea6-394a-4538-b186-efaf14574ccb" (UID: "5a467ea6-394a-4538-b186-efaf14574ccb"). InnerVolumeSpecName "kube-api-access-6mffn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:16.943156 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:16.943019    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mffn\" (UniqueName: \"kubernetes.io/projected/5a467ea6-394a-4538-b186-efaf14574ccb-kube-api-access-6mffn\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:30:16.943156 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:16.943066    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a467ea6-394a-4538-b186-efaf14574ccb-maas-api-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:30:17.359988 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:17.359960    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq"
Oct 30 01:30:17.360185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:17.359953    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq" event={"ID":"5a467ea6-394a-4538-b186-efaf14574ccb","Type":"ContainerDied","Data":"86c437fbf3da54f34c0dbef8577024e011bbcbf412fb277bf36c6108fc4a3dc7"}
Oct 30 01:30:17.360185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:17.360095    6099 scope.go:122] "RemoveContainer" containerID="662d4b9889b09bcb9a6fee92ccb4a923f3757b502b0cd133763a0e74d1b25735"
Oct 30 01:30:17.432337 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:17.431192    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq"]
Oct 30 01:30:17.443863 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:17.443715    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-jn7tq"]
Oct 30 01:30:18.259767 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:18.259736    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a467ea6-394a-4538-b186-efaf14574ccb" path="/var/lib/kubelet/pods/5a467ea6-394a-4538-b186-efaf14574ccb/volumes"
Oct 30 01:30:19.951389 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:19.951358    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx"]
Oct 30 01:30:19.951817 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:19.951785    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5a467ea6-394a-4538-b186-efaf14574ccb" containerName="cleanup"
Oct 30 01:30:19.951817 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:19.951797    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a467ea6-394a-4538-b186-efaf14574ccb" containerName="cleanup"
Oct 30 01:30:19.951817 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:19.951810    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5a467ea6-394a-4538-b186-efaf14574ccb" containerName="cleanup"
Oct 30 01:30:19.951817 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:19.951815    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a467ea6-394a-4538-b186-efaf14574ccb" containerName="cleanup"
Oct 30 01:30:19.951983 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:19.951902    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a467ea6-394a-4538-b186-efaf14574ccb" containerName="cleanup"
Oct 30 01:30:19.951983 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:19.951913    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a467ea6-394a-4538-b186-efaf14574ccb" containerName="cleanup"
Oct 30 01:30:19.951983 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:19.951920    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a467ea6-394a-4538-b186-efaf14574ccb" containerName="cleanup"
Oct 30 01:30:19.956570 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:19.956553    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx"
Oct 30 01:30:19.973020 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:19.972987    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:30:19.984910 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:19.984857    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:30:19.985069 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:19.984950    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:30:19.985327 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:19.985107    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:30:19.987477 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:19.987454    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx"]
Oct 30 01:30:20.074788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:20.074754    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-trzfx\" (UID: \"2b2039c6-8ff3-42a9-bc9d-b5e85cb76177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx"
Oct 30 01:30:20.074788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:20.074788    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgj4z\" (UniqueName: \"kubernetes.io/projected/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-kube-api-access-bgj4z\") pod \"maas-api-revoke-sub-0b7e8edb-trzfx\" (UID: \"2b2039c6-8ff3-42a9-bc9d-b5e85cb76177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx"
Oct 30 01:30:20.075015 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:20.074898    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-trzfx\" (UID: \"2b2039c6-8ff3-42a9-bc9d-b5e85cb76177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx"
Oct 30 01:30:20.175949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:20.175920    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-trzfx\" (UID: \"2b2039c6-8ff3-42a9-bc9d-b5e85cb76177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx"
Oct 30 01:30:20.176110 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:20.175952    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgj4z\" (UniqueName: \"kubernetes.io/projected/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-kube-api-access-bgj4z\") pod \"maas-api-revoke-sub-0b7e8edb-trzfx\" (UID: \"2b2039c6-8ff3-42a9-bc9d-b5e85cb76177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx"
Oct 30 01:30:20.176110 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:20.176023    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-trzfx\" (UID: \"2b2039c6-8ff3-42a9-bc9d-b5e85cb76177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx"
Oct 30 01:30:20.176559 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:20.176541    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-trzfx\" (UID: \"2b2039c6-8ff3-42a9-bc9d-b5e85cb76177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx"
Oct 30 01:30:20.178353 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:20.178333    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-trzfx\" (UID: \"2b2039c6-8ff3-42a9-bc9d-b5e85cb76177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx"
Oct 30 01:30:20.199040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:20.199014    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgj4z\" (UniqueName: \"kubernetes.io/projected/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-kube-api-access-bgj4z\") pod \"maas-api-revoke-sub-0b7e8edb-trzfx\" (UID: \"2b2039c6-8ff3-42a9-bc9d-b5e85cb76177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx"
Oct 30 01:30:20.269952 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:20.269884    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx"
Oct 30 01:30:20.568795 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:20.568758    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx"]
Oct 30 01:30:20.569766 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:30:20.569738    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2039c6_8ff3_42a9_bc9d_b5e85cb76177.slice/crio-ea257a20335800e18e08efdea22bade8783e8ad44530fc649c96bf467d16a4f5 WatchSource:0}: Error finding container ea257a20335800e18e08efdea22bade8783e8ad44530fc649c96bf467d16a4f5: Status 404 returned error can't find the container with id ea257a20335800e18e08efdea22bade8783e8ad44530fc649c96bf467d16a4f5
Oct 30 01:30:21.385826 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:21.385791    6099 generic.go:356] "Generic (PLEG): container finished" podID="2b2039c6-8ff3-42a9-bc9d-b5e85cb76177" containerID="c49013c332719ef5143ff60c8a340ed923baeefa454ccadf1db3082f2b61e785" exitCode=0
Oct 30 01:30:21.386253 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:21.385877    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx" event={"ID":"2b2039c6-8ff3-42a9-bc9d-b5e85cb76177","Type":"ContainerDied","Data":"c49013c332719ef5143ff60c8a340ed923baeefa454ccadf1db3082f2b61e785"}
Oct 30 01:30:21.386253 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:21.385929    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx" event={"ID":"2b2039c6-8ff3-42a9-bc9d-b5e85cb76177","Type":"ContainerStarted","Data":"ea257a20335800e18e08efdea22bade8783e8ad44530fc649c96bf467d16a4f5"}
Oct 30 01:30:22.525613 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:22.525590    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx"
Oct 30 01:30:22.701578 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:22.701492    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-maas-api-openshift-ca\") pod \"2b2039c6-8ff3-42a9-bc9d-b5e85cb76177\" (UID: \"2b2039c6-8ff3-42a9-bc9d-b5e85cb76177\") "
Oct 30 01:30:22.701773 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:22.701587    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-maas-api-xks-ca\") pod \"2b2039c6-8ff3-42a9-bc9d-b5e85cb76177\" (UID: \"2b2039c6-8ff3-42a9-bc9d-b5e85cb76177\") "
Oct 30 01:30:22.701773 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:22.701752    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-kube-api-access-bgj4z\" (UniqueName: \"kubernetes.io/projected/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-kube-api-access-bgj4z\") pod \"2b2039c6-8ff3-42a9-bc9d-b5e85cb76177\" (UID: \"2b2039c6-8ff3-42a9-bc9d-b5e85cb76177\") "
Oct 30 01:30:22.701916 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:22.701817    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-maas-api-openshift-ca" pod "2b2039c6-8ff3-42a9-bc9d-b5e85cb76177" (UID: "2b2039c6-8ff3-42a9-bc9d-b5e85cb76177"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:22.702083 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:22.702065    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:30:22.703606 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:22.703573    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-maas-api-xks-ca" pod "2b2039c6-8ff3-42a9-bc9d-b5e85cb76177" (UID: "2b2039c6-8ff3-42a9-bc9d-b5e85cb76177"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:22.703730 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:22.703676    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-kube-api-access-bgj4z" pod "2b2039c6-8ff3-42a9-bc9d-b5e85cb76177" (UID: "2b2039c6-8ff3-42a9-bc9d-b5e85cb76177"). InnerVolumeSpecName "kube-api-access-bgj4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:22.803143 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:22.803111    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgj4z\" (UniqueName: \"kubernetes.io/projected/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-kube-api-access-bgj4z\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:30:22.803143 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:22.803138    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:30:23.397448 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:23.397416    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx"
Oct 30 01:30:23.397622 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:23.397425    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx" event={"ID":"2b2039c6-8ff3-42a9-bc9d-b5e85cb76177","Type":"ContainerDied","Data":"ea257a20335800e18e08efdea22bade8783e8ad44530fc649c96bf467d16a4f5"}
Oct 30 01:30:23.397622 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:23.397541    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea257a20335800e18e08efdea22bade8783e8ad44530fc649c96bf467d16a4f5"
Oct 30 01:30:30.691416 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.691382    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc"]
Oct 30 01:30:30.691847 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.691833    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5a467ea6-394a-4538-b186-efaf14574ccb" containerName="cleanup"
Oct 30 01:30:30.691893 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.691849    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a467ea6-394a-4538-b186-efaf14574ccb" containerName="cleanup"
Oct 30 01:30:30.691893 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.691864    6099 cpu_manager.go:414] "containerMap: removing container" podUID="2b2039c6-8ff3-42a9-bc9d-b5e85cb76177" containerName="revoke-keys"
Oct 30 01:30:30.691893 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.691871    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2039c6-8ff3-42a9-bc9d-b5e85cb76177" containerName="revoke-keys"
Oct 30 01:30:30.692008 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.691972    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2039c6-8ff3-42a9-bc9d-b5e85cb76177" containerName="revoke-keys"
Oct 30 01:30:30.695432 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.695410    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc"
Oct 30 01:30:30.712341 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.712317    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:30:30.724003 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.723977    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:30:30.727219 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.727190    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc"]
Oct 30 01:30:30.738917 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.738896    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:30:30.739072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.738989    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:30:30.774031 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.773993    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4xdq\" (UniqueName: \"kubernetes.io/projected/862b8d7d-9f17-4032-b734-604f1573d515-kube-api-access-c4xdq\") pod \"maas-api-revoke-sub-3d4156f1-xsxvc\" (UID: \"862b8d7d-9f17-4032-b734-604f1573d515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc"
Oct 30 01:30:30.774226 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.774143    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862b8d7d-9f17-4032-b734-604f1573d515-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xsxvc\" (UID: \"862b8d7d-9f17-4032-b734-604f1573d515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc"
Oct 30 01:30:30.774226 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.774187    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862b8d7d-9f17-4032-b734-604f1573d515-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xsxvc\" (UID: \"862b8d7d-9f17-4032-b734-604f1573d515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc"
Oct 30 01:30:30.875291 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.875260    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862b8d7d-9f17-4032-b734-604f1573d515-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xsxvc\" (UID: \"862b8d7d-9f17-4032-b734-604f1573d515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc"
Oct 30 01:30:30.875509 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.875297    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4xdq\" (UniqueName: \"kubernetes.io/projected/862b8d7d-9f17-4032-b734-604f1573d515-kube-api-access-c4xdq\") pod \"maas-api-revoke-sub-3d4156f1-xsxvc\" (UID: \"862b8d7d-9f17-4032-b734-604f1573d515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc"
Oct 30 01:30:30.875509 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.875394    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862b8d7d-9f17-4032-b734-604f1573d515-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xsxvc\" (UID: \"862b8d7d-9f17-4032-b734-604f1573d515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc"
Oct 30 01:30:30.876019 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.875995    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862b8d7d-9f17-4032-b734-604f1573d515-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xsxvc\" (UID: \"862b8d7d-9f17-4032-b734-604f1573d515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc"
Oct 30 01:30:30.877642 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.877620    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862b8d7d-9f17-4032-b734-604f1573d515-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xsxvc\" (UID: \"862b8d7d-9f17-4032-b734-604f1573d515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc"
Oct 30 01:30:30.898417 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:30.898389    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4xdq\" (UniqueName: \"kubernetes.io/projected/862b8d7d-9f17-4032-b734-604f1573d515-kube-api-access-c4xdq\") pod \"maas-api-revoke-sub-3d4156f1-xsxvc\" (UID: \"862b8d7d-9f17-4032-b734-604f1573d515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc"
Oct 30 01:30:31.012961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:31.012893    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc"
Oct 30 01:30:31.206131 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:31.206095    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:30:31.320458 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:31.320434    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc"]
Oct 30 01:30:31.322005 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:30:31.321979    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862b8d7d_9f17_4032_b734_604f1573d515.slice/crio-e996863dd7ab2a06532f1b4bbe85cbe74c5aa54fd98269fb9a7a685969ad936d WatchSource:0}: Error finding container e996863dd7ab2a06532f1b4bbe85cbe74c5aa54fd98269fb9a7a685969ad936d: Status 404 returned error can't find the container with id e996863dd7ab2a06532f1b4bbe85cbe74c5aa54fd98269fb9a7a685969ad936d
Oct 30 01:30:31.450505 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:31.450472    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc" event={"ID":"862b8d7d-9f17-4032-b734-604f1573d515","Type":"ContainerStarted","Data":"e996863dd7ab2a06532f1b4bbe85cbe74c5aa54fd98269fb9a7a685969ad936d"}
Oct 30 01:30:32.458191 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:32.458100    6099 generic.go:356] "Generic (PLEG): container finished" podID="862b8d7d-9f17-4032-b734-604f1573d515" containerID="d4d9af4563606a0d16b1cfddb46cdcfdba10c960a10ab2f50dc9d7923e6e5c2b" exitCode=0
Oct 30 01:30:32.458191 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:32.458139    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc" event={"ID":"862b8d7d-9f17-4032-b734-604f1573d515","Type":"ContainerDied","Data":"d4d9af4563606a0d16b1cfddb46cdcfdba10c960a10ab2f50dc9d7923e6e5c2b"}
Oct 30 01:30:33.597392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:33.597366    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc"
Oct 30 01:30:33.599381 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:33.599360    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/862b8d7d-9f17-4032-b734-604f1573d515-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862b8d7d-9f17-4032-b734-604f1573d515-maas-api-openshift-ca\") pod \"862b8d7d-9f17-4032-b734-604f1573d515\" (UID: \"862b8d7d-9f17-4032-b734-604f1573d515\") "
Oct 30 01:30:33.599444 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:33.599403    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/862b8d7d-9f17-4032-b734-604f1573d515-kube-api-access-c4xdq\" (UniqueName: \"kubernetes.io/projected/862b8d7d-9f17-4032-b734-604f1573d515-kube-api-access-c4xdq\") pod \"862b8d7d-9f17-4032-b734-604f1573d515\" (UID: \"862b8d7d-9f17-4032-b734-604f1573d515\") "
Oct 30 01:30:33.599444 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:33.599430    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/862b8d7d-9f17-4032-b734-604f1573d515-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862b8d7d-9f17-4032-b734-604f1573d515-maas-api-xks-ca\") pod \"862b8d7d-9f17-4032-b734-604f1573d515\" (UID: \"862b8d7d-9f17-4032-b734-604f1573d515\") "
Oct 30 01:30:33.599670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:33.599650    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/862b8d7d-9f17-4032-b734-604f1573d515-maas-api-openshift-ca" pod "862b8d7d-9f17-4032-b734-604f1573d515" (UID: "862b8d7d-9f17-4032-b734-604f1573d515"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:33.601450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:33.601423    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862b8d7d-9f17-4032-b734-604f1573d515-kube-api-access-c4xdq" pod "862b8d7d-9f17-4032-b734-604f1573d515" (UID: "862b8d7d-9f17-4032-b734-604f1573d515"). InnerVolumeSpecName "kube-api-access-c4xdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:33.601513 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:33.601462    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/862b8d7d-9f17-4032-b734-604f1573d515-maas-api-xks-ca" pod "862b8d7d-9f17-4032-b734-604f1573d515" (UID: "862b8d7d-9f17-4032-b734-604f1573d515"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:33.700377 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:33.700345    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862b8d7d-9f17-4032-b734-604f1573d515-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:30:33.700377 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:33.700371    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4xdq\" (UniqueName: \"kubernetes.io/projected/862b8d7d-9f17-4032-b734-604f1573d515-kube-api-access-c4xdq\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:30:33.700377 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:33.700380    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862b8d7d-9f17-4032-b734-604f1573d515-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:30:34.470729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:34.470678    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc" event={"ID":"862b8d7d-9f17-4032-b734-604f1573d515","Type":"ContainerDied","Data":"e996863dd7ab2a06532f1b4bbe85cbe74c5aa54fd98269fb9a7a685969ad936d"}
Oct 30 01:30:34.470729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:34.470727    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc"
Oct 30 01:30:34.470993 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:34.470739    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e996863dd7ab2a06532f1b4bbe85cbe74c5aa54fd98269fb9a7a685969ad936d"
Oct 30 01:30:41.935758 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:41.935720    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:30:46.628153 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:30:46.628113    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:30:47.108021 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:47.107983    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:30:51.991667 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:51.991631    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc"]
Oct 30 01:30:52.003670 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.003635    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xsxvc"]
Oct 30 01:30:52.037564 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.037526    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22"]
Oct 30 01:30:52.037973 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.037959    6099 cpu_manager.go:414] "containerMap: removing container" podUID="862b8d7d-9f17-4032-b734-604f1573d515" containerName="revoke-keys"
Oct 30 01:30:52.038034 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.037974    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="862b8d7d-9f17-4032-b734-604f1573d515" containerName="revoke-keys"
Oct 30 01:30:52.038082 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.038072    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="862b8d7d-9f17-4032-b734-604f1573d515" containerName="revoke-keys"
Oct 30 01:30:52.041489 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.041468    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22"
Oct 30 01:30:52.058823 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.058690    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:30:52.058983 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.058879    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:30:52.058983 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.058956    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:30:52.059102 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.059008    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:30:52.061092 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.061067    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22"]
Oct 30 01:30:52.107220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.107188    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2hxn\" (UniqueName: \"kubernetes.io/projected/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-kube-api-access-w2hxn\") pod \"maas-api-revoke-sub-3d4156f1-gsz22\" (UID: \"2f1a1a5c-52d6-4875-ad92-8c31f7a06178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22"
Oct 30 01:30:52.107375 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.107225    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gsz22\" (UID: \"2f1a1a5c-52d6-4875-ad92-8c31f7a06178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22"
Oct 30 01:30:52.107375 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.107312    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gsz22\" (UID: \"2f1a1a5c-52d6-4875-ad92-8c31f7a06178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22"
Oct 30 01:30:52.207837 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.207803    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gsz22\" (UID: \"2f1a1a5c-52d6-4875-ad92-8c31f7a06178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22"
Oct 30 01:30:52.208027 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.207943    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2hxn\" (UniqueName: \"kubernetes.io/projected/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-kube-api-access-w2hxn\") pod \"maas-api-revoke-sub-3d4156f1-gsz22\" (UID: \"2f1a1a5c-52d6-4875-ad92-8c31f7a06178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22"
Oct 30 01:30:52.208027 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.207969    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gsz22\" (UID: \"2f1a1a5c-52d6-4875-ad92-8c31f7a06178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22"
Oct 30 01:30:52.208496 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.208458    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gsz22\" (UID: \"2f1a1a5c-52d6-4875-ad92-8c31f7a06178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22"
Oct 30 01:30:52.210233 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.210207    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gsz22\" (UID: \"2f1a1a5c-52d6-4875-ad92-8c31f7a06178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22"
Oct 30 01:30:52.230872 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.230849    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2hxn\" (UniqueName: \"kubernetes.io/projected/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-kube-api-access-w2hxn\") pod \"maas-api-revoke-sub-3d4156f1-gsz22\" (UID: \"2f1a1a5c-52d6-4875-ad92-8c31f7a06178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22"
Oct 30 01:30:52.260306 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.260245    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="862b8d7d-9f17-4032-b734-604f1573d515" path="/var/lib/kubelet/pods/862b8d7d-9f17-4032-b734-604f1573d515/volumes"
Oct 30 01:30:52.354893 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.354870    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22"
Oct 30 01:30:52.651179 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:52.651146    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22"]
Oct 30 01:30:52.652756 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:30:52.652723    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1a1a5c_52d6_4875_ad92_8c31f7a06178.slice/crio-f6a41cebc2fc6672b8cad5dd94220709204dd274df0cf238022b4497b722465e WatchSource:0}: Error finding container f6a41cebc2fc6672b8cad5dd94220709204dd274df0cf238022b4497b722465e: Status 404 returned error can't find the container with id f6a41cebc2fc6672b8cad5dd94220709204dd274df0cf238022b4497b722465e
Oct 30 01:30:53.595382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:53.595348    6099 generic.go:356] "Generic (PLEG): container finished" podID="2f1a1a5c-52d6-4875-ad92-8c31f7a06178" containerID="d76be53431b1c7fbe68dac7e545e42c4a380b3faede222087db97f74fa28ffbd" exitCode=0
Oct 30 01:30:53.595822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:53.595438    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22" event={"ID":"2f1a1a5c-52d6-4875-ad92-8c31f7a06178","Type":"ContainerDied","Data":"d76be53431b1c7fbe68dac7e545e42c4a380b3faede222087db97f74fa28ffbd"}
Oct 30 01:30:53.595822 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:53.595506    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22" event={"ID":"2f1a1a5c-52d6-4875-ad92-8c31f7a06178","Type":"ContainerStarted","Data":"f6a41cebc2fc6672b8cad5dd94220709204dd274df0cf238022b4497b722465e"}
Oct 30 01:30:54.729910 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:54.729887    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22"
Oct 30 01:30:54.833787 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:54.833760    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-maas-api-openshift-ca\") pod \"2f1a1a5c-52d6-4875-ad92-8c31f7a06178\" (UID: \"2f1a1a5c-52d6-4875-ad92-8c31f7a06178\") "
Oct 30 01:30:54.833960 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:54.833941    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-maas-api-xks-ca\") pod \"2f1a1a5c-52d6-4875-ad92-8c31f7a06178\" (UID: \"2f1a1a5c-52d6-4875-ad92-8c31f7a06178\") "
Oct 30 01:30:54.834010 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:54.833972    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-kube-api-access-w2hxn\" (UniqueName: \"kubernetes.io/projected/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-kube-api-access-w2hxn\") pod \"2f1a1a5c-52d6-4875-ad92-8c31f7a06178\" (UID: \"2f1a1a5c-52d6-4875-ad92-8c31f7a06178\") "
Oct 30 01:30:54.834163 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:54.834134    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-maas-api-openshift-ca" pod "2f1a1a5c-52d6-4875-ad92-8c31f7a06178" (UID: "2f1a1a5c-52d6-4875-ad92-8c31f7a06178"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:54.834225 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:54.834212    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:30:54.836010 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:54.835983    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-maas-api-xks-ca" pod "2f1a1a5c-52d6-4875-ad92-8c31f7a06178" (UID: "2f1a1a5c-52d6-4875-ad92-8c31f7a06178"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:54.836010 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:54.835995    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-kube-api-access-w2hxn" pod "2f1a1a5c-52d6-4875-ad92-8c31f7a06178" (UID: "2f1a1a5c-52d6-4875-ad92-8c31f7a06178"). InnerVolumeSpecName "kube-api-access-w2hxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:54.935069 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:54.935000    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:30:54.935069 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:54.935028    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2hxn\" (UniqueName: \"kubernetes.io/projected/2f1a1a5c-52d6-4875-ad92-8c31f7a06178-kube-api-access-w2hxn\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:30:55.029771 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:55.029737    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g"]
Oct 30 01:30:55.031556 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:55.031530    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3f9826e-khr2g"]
Oct 30 01:30:55.041303 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:55.041273    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz"]
Oct 30 01:30:55.043026 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:55.043000    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4609f84-h89sz"]
Oct 30 01:30:55.608830 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:55.608801    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22"
Oct 30 01:30:55.608830 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:55.608812    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22" event={"ID":"2f1a1a5c-52d6-4875-ad92-8c31f7a06178","Type":"ContainerDied","Data":"f6a41cebc2fc6672b8cad5dd94220709204dd274df0cf238022b4497b722465e"}
Oct 30 01:30:55.609057 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:55.608856    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6a41cebc2fc6672b8cad5dd94220709204dd274df0cf238022b4497b722465e"
Oct 30 01:30:56.260005 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:56.259973    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac579e0b-c335-44da-82d1-760d7066ccc7" path="/var/lib/kubelet/pods/ac579e0b-c335-44da-82d1-760d7066ccc7/volumes"
Oct 30 01:30:56.260380 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:30:56.260367    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e515a259-784c-4778-af11-4a0088e3cef8" path="/var/lib/kubelet/pods/e515a259-784c-4778-af11-4a0088e3cef8/volumes"
Oct 30 01:31:04.025893 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:04.025853    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:31:09.043572 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:09.043536    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:31:14.030173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:14.030136    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk"]
Oct 30 01:31:14.042213 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:14.042185    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee3ff2c1-2qqhk"]
Oct 30 01:31:14.260400 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:14.260368    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3420de78-6b4a-4fa8-b78d-b9129cabd93f" path="/var/lib/kubelet/pods/3420de78-6b4a-4fa8-b78d-b9129cabd93f/volumes"
Oct 30 01:31:15.620410 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.620375    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22"]
Oct 30 01:31:15.632048 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.632023    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gsz22"]
Oct 30 01:31:15.672339 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.672311    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt"]
Oct 30 01:31:15.672761 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.672746    6099 cpu_manager.go:414] "containerMap: removing container" podUID="2f1a1a5c-52d6-4875-ad92-8c31f7a06178" containerName="revoke-keys"
Oct 30 01:31:15.672821 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.672763    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1a1a5c-52d6-4875-ad92-8c31f7a06178" containerName="revoke-keys"
Oct 30 01:31:15.672866 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.672851    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1a1a5c-52d6-4875-ad92-8c31f7a06178" containerName="revoke-keys"
Oct 30 01:31:15.676252 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.676233    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt"
Oct 30 01:31:15.693745 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.693720    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:31:15.693904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.693741    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:31:15.693904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.693755    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:31:15.694053 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.693775    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:31:15.696070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.696049    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt"]
Oct 30 01:31:15.702964 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.702938    6099 scope.go:122] "RemoveContainer" containerID="77128293d73b3e8d364ae5d21b598853a47dbc12c286293ad33af07826b3aea8"
Oct 30 01:31:15.712197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.712175    6099 scope.go:122] "RemoveContainer" containerID="e097cb112793aa5b06fbd2f11b89c18d802a388902d923ffef184bbc86a1c9ae"
Oct 30 01:31:15.723104 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.723080    6099 scope.go:122] "RemoveContainer" containerID="4ca5e5a144ee25f920fa4ebb88efda71ba202b289091619d207897614bfff917"
Oct 30 01:31:15.731622 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.731602    6099 scope.go:122] "RemoveContainer" containerID="aad4d78276f671761a0763593022fce4296673fbbe9839f1e686688bcd10a478"
Oct 30 01:31:15.739336 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.739312    6099 scope.go:122] "RemoveContainer" containerID="ccb845bf03ebdf4918654d6745532b1d1aa0006c95dfa11a7ebdca4bbb17d39a"
Oct 30 01:31:15.837235 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.837205    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a11381-ea09-443f-90b5-89090cd60a7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7cldt\" (UID: \"55a11381-ea09-443f-90b5-89090cd60a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt"
Oct 30 01:31:15.837351 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.837328    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzbq5\" (UniqueName: \"kubernetes.io/projected/55a11381-ea09-443f-90b5-89090cd60a7d-kube-api-access-rzbq5\") pod \"maas-api-revoke-sub-3d4156f1-7cldt\" (UID: \"55a11381-ea09-443f-90b5-89090cd60a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt"
Oct 30 01:31:15.837401 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.837371    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a11381-ea09-443f-90b5-89090cd60a7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7cldt\" (UID: \"55a11381-ea09-443f-90b5-89090cd60a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt"
Oct 30 01:31:15.938834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.938767    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzbq5\" (UniqueName: \"kubernetes.io/projected/55a11381-ea09-443f-90b5-89090cd60a7d-kube-api-access-rzbq5\") pod \"maas-api-revoke-sub-3d4156f1-7cldt\" (UID: \"55a11381-ea09-443f-90b5-89090cd60a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt"
Oct 30 01:31:15.938834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.938825    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a11381-ea09-443f-90b5-89090cd60a7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7cldt\" (UID: \"55a11381-ea09-443f-90b5-89090cd60a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt"
Oct 30 01:31:15.939034 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.938869    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a11381-ea09-443f-90b5-89090cd60a7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7cldt\" (UID: \"55a11381-ea09-443f-90b5-89090cd60a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt"
Oct 30 01:31:15.939533 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.939512    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a11381-ea09-443f-90b5-89090cd60a7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7cldt\" (UID: \"55a11381-ea09-443f-90b5-89090cd60a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt"
Oct 30 01:31:15.941148 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.941131    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a11381-ea09-443f-90b5-89090cd60a7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7cldt\" (UID: \"55a11381-ea09-443f-90b5-89090cd60a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt"
Oct 30 01:31:15.961545 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.961522    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzbq5\" (UniqueName: \"kubernetes.io/projected/55a11381-ea09-443f-90b5-89090cd60a7d-kube-api-access-rzbq5\") pod \"maas-api-revoke-sub-3d4156f1-7cldt\" (UID: \"55a11381-ea09-443f-90b5-89090cd60a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt"
Oct 30 01:31:15.989252 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:15.989221    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt"
Oct 30 01:31:16.259612 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:16.259540    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1a1a5c-52d6-4875-ad92-8c31f7a06178" path="/var/lib/kubelet/pods/2f1a1a5c-52d6-4875-ad92-8c31f7a06178/volumes"
Oct 30 01:31:16.287769 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:16.287744    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt"]
Oct 30 01:31:16.288717 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:31:16.288674    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a11381_ea09_443f_90b5_89090cd60a7d.slice/crio-bc95acd2effd1f995c22c20d52c66b66f1c123315daf5d55cb726482f4e30eb2 WatchSource:0}: Error finding container bc95acd2effd1f995c22c20d52c66b66f1c123315daf5d55cb726482f4e30eb2: Status 404 returned error can't find the container with id bc95acd2effd1f995c22c20d52c66b66f1c123315daf5d55cb726482f4e30eb2
Oct 30 01:31:16.628801 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:31:16.628768    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:31:16.733333 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:16.733295    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt" event={"ID":"55a11381-ea09-443f-90b5-89090cd60a7d","Type":"ContainerStarted","Data":"091ca9988b3888704afadf32f8d60b992b795ce99cb7ea21a534744acc64541c"}
Oct 30 01:31:16.733521 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:16.733359    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt" event={"ID":"55a11381-ea09-443f-90b5-89090cd60a7d","Type":"ContainerStarted","Data":"bc95acd2effd1f995c22c20d52c66b66f1c123315daf5d55cb726482f4e30eb2"}
Oct 30 01:31:16.773122 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:16.773074    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt" podStartSLOduration=1.773054713 podStartE2EDuration="1.773054713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:16.772544398 +0000 UTC m=+3370.629452146" watchObservedRunningTime="2026-10-30 01:31:16.773054713 +0000 UTC m=+3370.629962469"
Oct 30 01:31:17.740615 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:17.740582    6099 generic.go:356] "Generic (PLEG): container finished" podID="55a11381-ea09-443f-90b5-89090cd60a7d" containerID="091ca9988b3888704afadf32f8d60b992b795ce99cb7ea21a534744acc64541c" exitCode=0
Oct 30 01:31:17.741218 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:17.740667    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt" event={"ID":"55a11381-ea09-443f-90b5-89090cd60a7d","Type":"ContainerDied","Data":"091ca9988b3888704afadf32f8d60b992b795ce99cb7ea21a534744acc64541c"}
Oct 30 01:31:18.882537 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:18.882507    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt"
Oct 30 01:31:18.969348 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:18.969317    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55a11381-ea09-443f-90b5-89090cd60a7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a11381-ea09-443f-90b5-89090cd60a7d-maas-api-xks-ca\") pod \"55a11381-ea09-443f-90b5-89090cd60a7d\" (UID: \"55a11381-ea09-443f-90b5-89090cd60a7d\") "
Oct 30 01:31:18.969535 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:18.969361    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a11381-ea09-443f-90b5-89090cd60a7d-kube-api-access-rzbq5\" (UniqueName: \"kubernetes.io/projected/55a11381-ea09-443f-90b5-89090cd60a7d-kube-api-access-rzbq5\") pod \"55a11381-ea09-443f-90b5-89090cd60a7d\" (UID: \"55a11381-ea09-443f-90b5-89090cd60a7d\") "
Oct 30 01:31:18.969535 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:18.969384    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55a11381-ea09-443f-90b5-89090cd60a7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a11381-ea09-443f-90b5-89090cd60a7d-maas-api-openshift-ca\") pod \"55a11381-ea09-443f-90b5-89090cd60a7d\" (UID: \"55a11381-ea09-443f-90b5-89090cd60a7d\") "
Oct 30 01:31:18.969795 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:18.969772    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a11381-ea09-443f-90b5-89090cd60a7d-maas-api-openshift-ca" pod "55a11381-ea09-443f-90b5-89090cd60a7d" (UID: "55a11381-ea09-443f-90b5-89090cd60a7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:18.971564 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:18.971532    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a11381-ea09-443f-90b5-89090cd60a7d-kube-api-access-rzbq5" pod "55a11381-ea09-443f-90b5-89090cd60a7d" (UID: "55a11381-ea09-443f-90b5-89090cd60a7d"). InnerVolumeSpecName "kube-api-access-rzbq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:18.971691 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:18.971600    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55a11381-ea09-443f-90b5-89090cd60a7d-maas-api-xks-ca" pod "55a11381-ea09-443f-90b5-89090cd60a7d" (UID: "55a11381-ea09-443f-90b5-89090cd60a7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:31:19.070811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:19.070779    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a11381-ea09-443f-90b5-89090cd60a7d-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:31:19.070811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:19.070807    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzbq5\" (UniqueName: \"kubernetes.io/projected/55a11381-ea09-443f-90b5-89090cd60a7d-kube-api-access-rzbq5\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:31:19.070811 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:19.070818    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a11381-ea09-443f-90b5-89090cd60a7d-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:31:19.752729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:19.752679    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt"
Oct 30 01:31:19.752729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:19.752698    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt" event={"ID":"55a11381-ea09-443f-90b5-89090cd60a7d","Type":"ContainerDied","Data":"bc95acd2effd1f995c22c20d52c66b66f1c123315daf5d55cb726482f4e30eb2"}
Oct 30 01:31:19.752955 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:19.752764    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc95acd2effd1f995c22c20d52c66b66f1c123315daf5d55cb726482f4e30eb2"
Oct 30 01:31:27.770553 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:27.770513    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:31:32.833693 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:32.826843    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:31:42.917431 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:42.917388    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"]
Oct 30 01:31:42.917980 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:42.917877    6099 cpu_manager.go:414] "containerMap: removing container" podUID="55a11381-ea09-443f-90b5-89090cd60a7d" containerName="revoke-keys"
Oct 30 01:31:42.917980 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:42.917892    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a11381-ea09-443f-90b5-89090cd60a7d" containerName="revoke-keys"
Oct 30 01:31:42.918121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:42.918013    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a11381-ea09-443f-90b5-89090cd60a7d" containerName="revoke-keys"
Oct 30 01:31:42.921423 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:42.921401    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"
Oct 30 01:31:42.950989 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:42.950957    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ff1bf5-serving-cert\""
Oct 30 01:31:42.951208 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:42.951067    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 01:31:42.951208 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:42.951134    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:31:42.951414 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:42.951289    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:31:42.951414 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:42.951332    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k5bc5\""
Oct 30 01:31:42.953239 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:42.953214    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"]
Oct 30 01:31:43.002500 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:43.002466    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa82156-9e25-4254-81e7-76bf325a45f9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn\" (UID: \"3fa82156-9e25-4254-81e7-76bf325a45f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"
Oct 30 01:31:43.002679 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:43.002516    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa82156-9e25-4254-81e7-76bf325a45f9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn\" (UID: \"3fa82156-9e25-4254-81e7-76bf325a45f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"
Oct 30 01:31:43.002679 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:43.002657    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqs66\" (UniqueName: \"kubernetes.io/projected/3fa82156-9e25-4254-81e7-76bf325a45f9-kube-api-access-bqs66\") pod \"maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn\" (UID: \"3fa82156-9e25-4254-81e7-76bf325a45f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"
Oct 30 01:31:43.002798 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:43.002785    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa82156-9e25-4254-81e7-76bf325a45f9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn\" (UID: \"3fa82156-9e25-4254-81e7-76bf325a45f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"
Oct 30 01:31:43.103634 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:43.103601    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa82156-9e25-4254-81e7-76bf325a45f9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn\" (UID: \"3fa82156-9e25-4254-81e7-76bf325a45f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"
Oct 30 01:31:43.103848 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:43.103651    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa82156-9e25-4254-81e7-76bf325a45f9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn\" (UID: \"3fa82156-9e25-4254-81e7-76bf325a45f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"
Oct 30 01:31:43.103848 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:43.103698    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa82156-9e25-4254-81e7-76bf325a45f9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn\" (UID: \"3fa82156-9e25-4254-81e7-76bf325a45f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"
Oct 30 01:31:43.103848 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:43.103819    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqs66\" (UniqueName: \"kubernetes.io/projected/3fa82156-9e25-4254-81e7-76bf325a45f9-kube-api-access-bqs66\") pod \"maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn\" (UID: \"3fa82156-9e25-4254-81e7-76bf325a45f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"
Oct 30 01:31:43.104302 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:43.104270    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa82156-9e25-4254-81e7-76bf325a45f9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn\" (UID: \"3fa82156-9e25-4254-81e7-76bf325a45f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"
Oct 30 01:31:43.106307 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:43.106282    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa82156-9e25-4254-81e7-76bf325a45f9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn\" (UID: \"3fa82156-9e25-4254-81e7-76bf325a45f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"
Oct 30 01:31:43.106402 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:43.106282    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa82156-9e25-4254-81e7-76bf325a45f9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn\" (UID: \"3fa82156-9e25-4254-81e7-76bf325a45f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"
Oct 30 01:31:43.126552 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:43.126531    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqs66\" (UniqueName: \"kubernetes.io/projected/3fa82156-9e25-4254-81e7-76bf325a45f9-kube-api-access-bqs66\") pod \"maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn\" (UID: \"3fa82156-9e25-4254-81e7-76bf325a45f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"
Oct 30 01:31:43.239497 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:43.239410    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"
Oct 30 01:31:43.545102 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:43.545073    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"]
Oct 30 01:31:43.546454 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:31:43.546414    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa82156_9e25_4254_81e7_76bf325a45f9.slice/crio-8899b85599ff02510ec3ec87d13d39ba2ebfb8be10feadb3c810fd4d7738f5e3 WatchSource:0}: Error finding container 8899b85599ff02510ec3ec87d13d39ba2ebfb8be10feadb3c810fd4d7738f5e3: Status 404 returned error can't find the container with id 8899b85599ff02510ec3ec87d13d39ba2ebfb8be10feadb3c810fd4d7738f5e3
Oct 30 01:31:43.547908 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:43.547889    6099 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:31:43.884093 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:43.884011    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn" event={"ID":"3fa82156-9e25-4254-81e7-76bf325a45f9","Type":"ContainerStarted","Data":"8899b85599ff02510ec3ec87d13d39ba2ebfb8be10feadb3c810fd4d7738f5e3"}
Oct 30 01:31:44.890765 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:44.890729    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn" event={"ID":"3fa82156-9e25-4254-81e7-76bf325a45f9","Type":"ContainerStarted","Data":"4b9bc5cfa668637d9a2497d6b1e126556425b0cee0c66be36610c7bf12e62857"}
Oct 30 01:31:44.891276 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:44.890829    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"
Oct 30 01:31:44.935200 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:44.935145    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn" podStartSLOduration=2.517133473 podStartE2EDuration="2.935131862s" totalImagesPullingTime="417.998389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:44.93259051 +0000 UTC m=+3398.789498264" watchObservedRunningTime="2026-10-30 01:31:44.935131862 +0000 UTC m=+3398.792039617"
Oct 30 01:31:46.630883 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:31:46.630807    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:31:46.642257 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:46.642219    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:31:50.901776 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:50.901751    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"
Oct 30 01:31:59.542070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.542038    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"]
Oct 30 01:31:59.546232 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.546214    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.575581 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.575547    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ff1bf5\"/\"openshift-service-ca.crt\""
Oct 30 01:31:59.575581 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.575548    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ff1bf5\"/\"simulator-single-main-ff1bf5-kserve-self-signed-certs\""
Oct 30 01:31:59.575909 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.575671    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ff1bf5\"/\"default-dockercfg-4qhfv\""
Oct 30 01:31:59.575909 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.575781    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ff1bf5\"/\"kube-root-ca.crt\""
Oct 30 01:31:59.576002 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.575960    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cb46ade-cfb4-4d1b-8c63-0054b7717583-tls-certs\") pod \"simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") " pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.576049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.576008    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-model-cache\") pod \"simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") " pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.576147 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.576125    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-home\") pod \"simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") " pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.576214 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.576192    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-dshm\") pod \"simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") " pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.576264 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.576221    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-tmp-dir\") pod \"simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") " pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.576361 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.576344    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5nrb\" (UniqueName: \"kubernetes.io/projected/5cb46ade-cfb4-4d1b-8c63-0054b7717583-kube-api-access-c5nrb\") pod \"simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") " pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.578934 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.578902    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"]
Oct 30 01:31:59.677304 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.677265    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cb46ade-cfb4-4d1b-8c63-0054b7717583-tls-certs\") pod \"simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") " pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.677495 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.677308    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-model-cache\") pod \"simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") " pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.677495 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.677377    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-home\") pod \"simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") " pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.677495 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.677397    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-dshm\") pod \"simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") " pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.677495 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.677419    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-tmp-dir\") pod \"simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") " pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.677495 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.677473    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5nrb\" (UniqueName: \"kubernetes.io/projected/5cb46ade-cfb4-4d1b-8c63-0054b7717583-kube-api-access-c5nrb\") pod \"simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") " pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.677855 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.677827    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-model-cache\") pod \"simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") " pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.678034 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.677938    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-home\") pod \"simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") " pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.678128 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.678110    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-tmp-dir\") pod \"simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") " pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.679883 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.679858    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-dshm\") pod \"simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") " pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.680522 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.680504    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cb46ade-cfb4-4d1b-8c63-0054b7717583-tls-certs\") pod \"simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") " pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.700511 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.700475    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5nrb\" (UniqueName: \"kubernetes.io/projected/5cb46ade-cfb4-4d1b-8c63-0054b7717583-kube-api-access-c5nrb\") pod \"simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") " pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:31:59.858884 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:31:59.858796    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:32:00.160518 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:00.160495    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"]
Oct 30 01:32:00.161914 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:32:00.161890    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cb46ade_cfb4_4d1b_8c63_0054b7717583.slice/crio-115a36ac0a0275aaa32e5e7ae80d191c8081c4c23b3cc3561613ffde078f3bbb WatchSource:0}: Error finding container 115a36ac0a0275aaa32e5e7ae80d191c8081c4c23b3cc3561613ffde078f3bbb: Status 404 returned error can't find the container with id 115a36ac0a0275aaa32e5e7ae80d191c8081c4c23b3cc3561613ffde078f3bbb
Oct 30 01:32:00.932598 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:00.932560    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:32:00.982270 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:00.982229    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc" event={"ID":"5cb46ade-cfb4-4d1b-8c63-0054b7717583","Type":"ContainerStarted","Data":"75b97b3cfb284b4a755bbc5998dd6fa9ad4efe3b43432ef4bf4aeadc06b1b6af"}
Oct 30 01:32:00.982474 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:00.982279    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc" event={"ID":"5cb46ade-cfb4-4d1b-8c63-0054b7717583","Type":"ContainerStarted","Data":"115a36ac0a0275aaa32e5e7ae80d191c8081c4c23b3cc3561613ffde078f3bbb"}
Oct 30 01:32:01.029987 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:01.029927    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc" podStartSLOduration=2.029911722 podStartE2EDuration="2.029911722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:01.028361372 +0000 UTC m=+3414.885269127" watchObservedRunningTime="2026-10-30 01:32:01.029911722 +0000 UTC m=+3414.886819474"
Oct 30 01:32:01.859267 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:01.859233    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:32:06.028273 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:06.028234    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc"]
Oct 30 01:32:06.039912 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:06.039885    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h4gdc"]
Oct 30 01:32:06.261556 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:06.261525    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86683fc9-cc0c-4228-9e0d-9ae63b326713" path="/var/lib/kubelet/pods/86683fc9-cc0c-4228-9e0d-9ae63b326713/volumes"
Oct 30 01:32:09.859531 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:09.859491    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:32:09.871699 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:09.871674    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:32:10.053461 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:10.053431    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:32:15.779654 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:15.779621    6099 scope.go:122] "RemoveContainer" containerID="abcefa3373930b9c6520a20aaa3c2157a7d102ad4a1b13c7566c9367d8024489"
Oct 30 01:32:15.791422 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:15.791396    6099 scope.go:122] "RemoveContainer" containerID="22dc558f479e628c6fab9700cc411df8ef6c462c6322f1e6cbc1be81146c20c6"
Oct 30 01:32:15.799510 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:15.799493    6099 scope.go:122] "RemoveContainer" containerID="5a551d5766a2ee643c05197a5d52d41b776881e6ca76abcb9555863b08584507"
Oct 30 01:32:15.807256 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:15.807238    6099 scope.go:122] "RemoveContainer" containerID="3ac0e43f7867db298c153a83538b361d4aea218f9abacf3416db762bd1c1bd14"
Oct 30 01:32:15.815750 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:15.815731    6099 scope.go:122] "RemoveContainer" containerID="9c3fd62156290e5dc06bedae110b000dd70d50521a166a8bb7906511a93392dc"
Oct 30 01:32:16.029615 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:16.029575    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:32:16.629261 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:32:16.629222    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:32:33.354758 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:33.354723    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:32:37.866485 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:37.866454    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"]
Oct 30 01:32:46.173119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:46.173079    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"]
Oct 30 01:32:46.173595 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:46.173277    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn" podUID="3fa82156-9e25-4254-81e7-76bf325a45f9" containerName="maas-api" containerID="cri-o://4b9bc5cfa668637d9a2497d6b1e126556425b0cee0c66be36610c7bf12e62857" gracePeriod=30
Oct 30 01:32:46.629977 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:32:46.629936    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:32:46.641381 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:46.641361    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"
Oct 30 01:32:46.758265 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:46.758194    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fa82156-9e25-4254-81e7-76bf325a45f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa82156-9e25-4254-81e7-76bf325a45f9-maas-api-tls\") pod \"3fa82156-9e25-4254-81e7-76bf325a45f9\" (UID: \"3fa82156-9e25-4254-81e7-76bf325a45f9\") "
Oct 30 01:32:46.758265 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:46.758246    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa82156-9e25-4254-81e7-76bf325a45f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa82156-9e25-4254-81e7-76bf325a45f9-openshift-service-ca\") pod \"3fa82156-9e25-4254-81e7-76bf325a45f9\" (UID: \"3fa82156-9e25-4254-81e7-76bf325a45f9\") "
Oct 30 01:32:46.758457 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:46.758326    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fa82156-9e25-4254-81e7-76bf325a45f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa82156-9e25-4254-81e7-76bf325a45f9-metrics-certs\") pod \"3fa82156-9e25-4254-81e7-76bf325a45f9\" (UID: \"3fa82156-9e25-4254-81e7-76bf325a45f9\") "
Oct 30 01:32:46.758457 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:46.758403    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa82156-9e25-4254-81e7-76bf325a45f9-kube-api-access-bqs66\" (UniqueName: \"kubernetes.io/projected/3fa82156-9e25-4254-81e7-76bf325a45f9-kube-api-access-bqs66\") pod \"3fa82156-9e25-4254-81e7-76bf325a45f9\" (UID: \"3fa82156-9e25-4254-81e7-76bf325a45f9\") "
Oct 30 01:32:46.758691 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:46.758665    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa82156-9e25-4254-81e7-76bf325a45f9-openshift-service-ca" pod "3fa82156-9e25-4254-81e7-76bf325a45f9" (UID: "3fa82156-9e25-4254-81e7-76bf325a45f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:32:46.760337 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:46.760310    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa82156-9e25-4254-81e7-76bf325a45f9-maas-api-tls" pod "3fa82156-9e25-4254-81e7-76bf325a45f9" (UID: "3fa82156-9e25-4254-81e7-76bf325a45f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:32:46.760464 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:46.760353    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa82156-9e25-4254-81e7-76bf325a45f9-metrics-certs" pod "3fa82156-9e25-4254-81e7-76bf325a45f9" (UID: "3fa82156-9e25-4254-81e7-76bf325a45f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:32:46.760464 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:46.760427    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa82156-9e25-4254-81e7-76bf325a45f9-kube-api-access-bqs66" pod "3fa82156-9e25-4254-81e7-76bf325a45f9" (UID: "3fa82156-9e25-4254-81e7-76bf325a45f9"). InnerVolumeSpecName "kube-api-access-bqs66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:46.859366 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:46.859336    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa82156-9e25-4254-81e7-76bf325a45f9-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:32:46.859366 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:46.859363    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqs66\" (UniqueName: \"kubernetes.io/projected/3fa82156-9e25-4254-81e7-76bf325a45f9-kube-api-access-bqs66\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:32:46.859571 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:46.859375    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa82156-9e25-4254-81e7-76bf325a45f9-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:32:46.859571 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:46.859386    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa82156-9e25-4254-81e7-76bf325a45f9-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:32:47.248315 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:47.248281    6099 generic.go:356] "Generic (PLEG): container finished" podID="3fa82156-9e25-4254-81e7-76bf325a45f9" containerID="4b9bc5cfa668637d9a2497d6b1e126556425b0cee0c66be36610c7bf12e62857" exitCode=0
Oct 30 01:32:47.248849 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:47.248340    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"
Oct 30 01:32:47.248849 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:47.248375    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn" event={"ID":"3fa82156-9e25-4254-81e7-76bf325a45f9","Type":"ContainerDied","Data":"4b9bc5cfa668637d9a2497d6b1e126556425b0cee0c66be36610c7bf12e62857"}
Oct 30 01:32:47.248849 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:47.248429    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn" event={"ID":"3fa82156-9e25-4254-81e7-76bf325a45f9","Type":"ContainerDied","Data":"8899b85599ff02510ec3ec87d13d39ba2ebfb8be10feadb3c810fd4d7738f5e3"}
Oct 30 01:32:47.248849 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:47.248452    6099 scope.go:122] "RemoveContainer" containerID="4b9bc5cfa668637d9a2497d6b1e126556425b0cee0c66be36610c7bf12e62857"
Oct 30 01:32:47.258556 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:47.258533    6099 scope.go:122] "RemoveContainer" containerID="4b9bc5cfa668637d9a2497d6b1e126556425b0cee0c66be36610c7bf12e62857"
Oct 30 01:32:47.258841 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:32:47.258821    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b9bc5cfa668637d9a2497d6b1e126556425b0cee0c66be36610c7bf12e62857\": container with ID starting with 4b9bc5cfa668637d9a2497d6b1e126556425b0cee0c66be36610c7bf12e62857 not found: ID does not exist" containerID="4b9bc5cfa668637d9a2497d6b1e126556425b0cee0c66be36610c7bf12e62857"
Oct 30 01:32:47.258932 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:47.258852    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9bc5cfa668637d9a2497d6b1e126556425b0cee0c66be36610c7bf12e62857"} err="failed to get container status \"4b9bc5cfa668637d9a2497d6b1e126556425b0cee0c66be36610c7bf12e62857\": rpc error: code = NotFound desc = could not find container \"4b9bc5cfa668637d9a2497d6b1e126556425b0cee0c66be36610c7bf12e62857\": container with ID starting with 4b9bc5cfa668637d9a2497d6b1e126556425b0cee0c66be36610c7bf12e62857 not found: ID does not exist"
Oct 30 01:32:47.298352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:47.298323    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"]
Oct 30 01:32:47.310371 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:47.310342    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ff1bf5-7cdf9bccdc-z6skn"]
Oct 30 01:32:48.259717 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:48.259682    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa82156-9e25-4254-81e7-76bf325a45f9" path="/var/lib/kubelet/pods/3fa82156-9e25-4254-81e7-76bf325a45f9/volumes"
Oct 30 01:32:48.462458 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:48.462426    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:32:52.886690 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:52.886647    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc" podUID="5cb46ade-cfb4-4d1b-8c63-0054b7717583" containerName="main" containerID="cri-o://75b97b3cfb284b4a755bbc5998dd6fa9ad4efe3b43432ef4bf4aeadc06b1b6af" gracePeriod=45
Oct 30 01:32:53.042984 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.042947    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc" podUID="5cb46ade-cfb4-4d1b-8c63-0054b7717583" containerName="main" probeResult="failure" output="Get \"https://10.134.1.95:8000/ready\": dial tcp 10.134.1.95:8000: connect: connection refused"
Oct 30 01:32:53.140300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.140244    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:32:53.223446 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.223412    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cb46ade-cfb4-4d1b-8c63-0054b7717583-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cb46ade-cfb4-4d1b-8c63-0054b7717583-tls-certs\") pod \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") "
Oct 30 01:32:53.223669 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.223464    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-home\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-home\") pod \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") "
Oct 30 01:32:53.223796 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.223760    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-tmp-dir\") pod \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") "
Oct 30 01:32:53.223862 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.223812    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-home" pod "5cb46ade-cfb4-4d1b-8c63-0054b7717583" (UID: "5cb46ade-cfb4-4d1b-8c63-0054b7717583"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:53.224022 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.223986    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-dshm\") pod \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") "
Oct 30 01:32:53.224205 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.224069    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb46ade-cfb4-4d1b-8c63-0054b7717583-kube-api-access-c5nrb\" (UniqueName: \"kubernetes.io/projected/5cb46ade-cfb4-4d1b-8c63-0054b7717583-kube-api-access-c5nrb\") pod \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") "
Oct 30 01:32:53.224205 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.224136    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-model-cache\") pod \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\" (UID: \"5cb46ade-cfb4-4d1b-8c63-0054b7717583\") "
Oct 30 01:32:53.224205 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.224137    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-tmp-dir" pod "5cb46ade-cfb4-4d1b-8c63-0054b7717583" (UID: "5cb46ade-cfb4-4d1b-8c63-0054b7717583"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:53.224536 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.224514    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:32:53.224674 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.224540    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:32:53.225462 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.225423    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-model-cache" pod "5cb46ade-cfb4-4d1b-8c63-0054b7717583" (UID: "5cb46ade-cfb4-4d1b-8c63-0054b7717583"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:53.226163 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.226137    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cb46ade-cfb4-4d1b-8c63-0054b7717583-tls-certs" pod "5cb46ade-cfb4-4d1b-8c63-0054b7717583" (UID: "5cb46ade-cfb4-4d1b-8c63-0054b7717583"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:32:53.226397 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.226376    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-dshm" pod "5cb46ade-cfb4-4d1b-8c63-0054b7717583" (UID: "5cb46ade-cfb4-4d1b-8c63-0054b7717583"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:32:53.226592 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.226568    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb46ade-cfb4-4d1b-8c63-0054b7717583-kube-api-access-c5nrb" pod "5cb46ade-cfb4-4d1b-8c63-0054b7717583" (UID: "5cb46ade-cfb4-4d1b-8c63-0054b7717583"). InnerVolumeSpecName "kube-api-access-c5nrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:53.289367 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.289335    6099 generic.go:356] "Generic (PLEG): container finished" podID="5cb46ade-cfb4-4d1b-8c63-0054b7717583" containerID="75b97b3cfb284b4a755bbc5998dd6fa9ad4efe3b43432ef4bf4aeadc06b1b6af" exitCode=0
Oct 30 01:32:53.289570 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.289382    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"
Oct 30 01:32:53.289570 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.289412    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc" event={"ID":"5cb46ade-cfb4-4d1b-8c63-0054b7717583","Type":"ContainerDied","Data":"75b97b3cfb284b4a755bbc5998dd6fa9ad4efe3b43432ef4bf4aeadc06b1b6af"}
Oct 30 01:32:53.289570 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.289463    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc" event={"ID":"5cb46ade-cfb4-4d1b-8c63-0054b7717583","Type":"ContainerDied","Data":"115a36ac0a0275aaa32e5e7ae80d191c8081c4c23b3cc3561613ffde078f3bbb"}
Oct 30 01:32:53.289570 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.289484    6099 scope.go:122] "RemoveContainer" containerID="75b97b3cfb284b4a755bbc5998dd6fa9ad4efe3b43432ef4bf4aeadc06b1b6af"
Oct 30 01:32:53.298921 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.298901    6099 scope.go:122] "RemoveContainer" containerID="75b97b3cfb284b4a755bbc5998dd6fa9ad4efe3b43432ef4bf4aeadc06b1b6af"
Oct 30 01:32:53.299173 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:32:53.299151    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b97b3cfb284b4a755bbc5998dd6fa9ad4efe3b43432ef4bf4aeadc06b1b6af\": container with ID starting with 75b97b3cfb284b4a755bbc5998dd6fa9ad4efe3b43432ef4bf4aeadc06b1b6af not found: ID does not exist" containerID="75b97b3cfb284b4a755bbc5998dd6fa9ad4efe3b43432ef4bf4aeadc06b1b6af"
Oct 30 01:32:53.299227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.299186    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b97b3cfb284b4a755bbc5998dd6fa9ad4efe3b43432ef4bf4aeadc06b1b6af"} err="failed to get container status \"75b97b3cfb284b4a755bbc5998dd6fa9ad4efe3b43432ef4bf4aeadc06b1b6af\": rpc error: code = NotFound desc = could not find container \"75b97b3cfb284b4a755bbc5998dd6fa9ad4efe3b43432ef4bf4aeadc06b1b6af\": container with ID starting with 75b97b3cfb284b4a755bbc5998dd6fa9ad4efe3b43432ef4bf4aeadc06b1b6af not found: ID does not exist"
Oct 30 01:32:53.325405 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.325384    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:32:53.325503 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.325407    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5nrb\" (UniqueName: \"kubernetes.io/projected/5cb46ade-cfb4-4d1b-8c63-0054b7717583-kube-api-access-c5nrb\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:32:53.325503 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.325417    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cb46ade-cfb4-4d1b-8c63-0054b7717583-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:32:53.325503 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.325427    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cb46ade-cfb4-4d1b-8c63-0054b7717583-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:32:53.338575 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.338555    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"]
Oct 30 01:32:53.342889 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:53.342870    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-ff1bf5/simulator-single-main-ff1bf5-kserve-78f9cfcc64-n55vc"]
Oct 30 01:32:54.028728 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:54.028684    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548"]
Oct 30 01:32:54.040521 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:54.040484    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-4s548"]
Oct 30 01:32:54.261608 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:54.261570    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cb46ade-cfb4-4d1b-8c63-0054b7717583" path="/var/lib/kubelet/pods/5cb46ade-cfb4-4d1b-8c63-0054b7717583/volumes"
Oct 30 01:32:54.262213 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:54.262190    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c5351f-6289-41cc-8a25-3dab37ca2913" path="/var/lib/kubelet/pods/a8c5351f-6289-41cc-8a25-3dab37ca2913/volumes"
Oct 30 01:32:55.128295 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.128260    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt"]
Oct 30 01:32:55.139803 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.139772    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7cldt"]
Oct 30 01:32:55.175785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.175757    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx"]
Oct 30 01:32:55.176243 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.176226    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5cb46ade-cfb4-4d1b-8c63-0054b7717583" containerName="main"
Oct 30 01:32:55.176300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.176245    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb46ade-cfb4-4d1b-8c63-0054b7717583" containerName="main"
Oct 30 01:32:55.176300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.176264    6099 cpu_manager.go:414] "containerMap: removing container" podUID="3fa82156-9e25-4254-81e7-76bf325a45f9" containerName="maas-api"
Oct 30 01:32:55.176300 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.176269    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa82156-9e25-4254-81e7-76bf325a45f9" containerName="maas-api"
Oct 30 01:32:55.176414 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.176345    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb46ade-cfb4-4d1b-8c63-0054b7717583" containerName="main"
Oct 30 01:32:55.176414 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.176355    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa82156-9e25-4254-81e7-76bf325a45f9" containerName="maas-api"
Oct 30 01:32:55.180769 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.180752    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx"
Oct 30 01:32:55.198074 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.198048    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:32:55.199065 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.198319    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:32:55.199065 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.198425    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:32:55.199065 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.198553    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:32:55.200500 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.200482    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx"]
Oct 30 01:32:55.346100 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.346064    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rnddx\" (UID: \"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx"
Oct 30 01:32:55.346306 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.346110    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zhlf\" (UniqueName: \"kubernetes.io/projected/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-kube-api-access-7zhlf\") pod \"maas-api-revoke-sub-3d4156f1-rnddx\" (UID: \"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx"
Oct 30 01:32:55.346385 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.346300    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rnddx\" (UID: \"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx"
Oct 30 01:32:55.447039 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.446952    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rnddx\" (UID: \"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx"
Oct 30 01:32:55.447039 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.446991    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zhlf\" (UniqueName: \"kubernetes.io/projected/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-kube-api-access-7zhlf\") pod \"maas-api-revoke-sub-3d4156f1-rnddx\" (UID: \"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx"
Oct 30 01:32:55.447293 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.447065    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rnddx\" (UID: \"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx"
Oct 30 01:32:55.447669 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.447648    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rnddx\" (UID: \"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx"
Oct 30 01:32:55.449352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.449332    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rnddx\" (UID: \"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx"
Oct 30 01:32:55.470156 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.470133    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zhlf\" (UniqueName: \"kubernetes.io/projected/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-kube-api-access-7zhlf\") pod \"maas-api-revoke-sub-3d4156f1-rnddx\" (UID: \"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx"
Oct 30 01:32:55.494036 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.494014    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx"
Oct 30 01:32:55.791691 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:55.791668    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx"]
Oct 30 01:32:55.793468 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:32:55.793440    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c063d2_e917_447f_94c7_1ff6cd9f1e7e.slice/crio-e5bc4adc4e90a6472ba156d713a1ce09c81428af7b603db5b790fbf0ab7caa2b WatchSource:0}: Error finding container e5bc4adc4e90a6472ba156d713a1ce09c81428af7b603db5b790fbf0ab7caa2b: Status 404 returned error can't find the container with id e5bc4adc4e90a6472ba156d713a1ce09c81428af7b603db5b790fbf0ab7caa2b
Oct 30 01:32:56.261009 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:56.260927    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a11381-ea09-443f-90b5-89090cd60a7d" path="/var/lib/kubelet/pods/55a11381-ea09-443f-90b5-89090cd60a7d/volumes"
Oct 30 01:32:56.309073 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:56.309029    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx" event={"ID":"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e","Type":"ContainerStarted","Data":"bab2e52fc5f2c01a889ef7ef0065df7e36e92c54b5c7ab80ca09ab65bb06d80e"}
Oct 30 01:32:56.309252 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:56.309092    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx" event={"ID":"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e","Type":"ContainerStarted","Data":"e5bc4adc4e90a6472ba156d713a1ce09c81428af7b603db5b790fbf0ab7caa2b"}
Oct 30 01:32:56.349507 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:56.349457    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx" podStartSLOduration=1.349444069 podStartE2EDuration="1.349444069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:56.348169318 +0000 UTC m=+3470.205077074" watchObservedRunningTime="2026-10-30 01:32:56.349444069 +0000 UTC m=+3470.206351877"
Oct 30 01:32:57.315544 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:57.315507    6099 generic.go:356] "Generic (PLEG): container finished" podID="f3c063d2-e917-447f-94c7-1ff6cd9f1e7e" containerID="bab2e52fc5f2c01a889ef7ef0065df7e36e92c54b5c7ab80ca09ab65bb06d80e" exitCode=0
Oct 30 01:32:57.315973 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:57.315553    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx" event={"ID":"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e","Type":"ContainerDied","Data":"bab2e52fc5f2c01a889ef7ef0065df7e36e92c54b5c7ab80ca09ab65bb06d80e"}
Oct 30 01:32:58.452163 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:58.452139    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx"
Oct 30 01:32:58.578451 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:58.578368    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-maas-api-xks-ca\") pod \"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e\" (UID: \"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e\") "
Oct 30 01:32:58.578637 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:58.578493    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-maas-api-openshift-ca\") pod \"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e\" (UID: \"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e\") "
Oct 30 01:32:58.578637 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:58.578564    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-kube-api-access-7zhlf\" (UniqueName: \"kubernetes.io/projected/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-kube-api-access-7zhlf\") pod \"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e\" (UID: \"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e\") "
Oct 30 01:32:58.578878 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:58.578850    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-maas-api-openshift-ca" pod "f3c063d2-e917-447f-94c7-1ff6cd9f1e7e" (UID: "f3c063d2-e917-447f-94c7-1ff6cd9f1e7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:32:58.580425 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:58.580404    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-maas-api-xks-ca" pod "f3c063d2-e917-447f-94c7-1ff6cd9f1e7e" (UID: "f3c063d2-e917-447f-94c7-1ff6cd9f1e7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:32:58.580536 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:58.580516    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-kube-api-access-7zhlf" pod "f3c063d2-e917-447f-94c7-1ff6cd9f1e7e" (UID: "f3c063d2-e917-447f-94c7-1ff6cd9f1e7e"). InnerVolumeSpecName "kube-api-access-7zhlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:58.679491 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:58.679464    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:32:58.679491 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:58.679488    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zhlf\" (UniqueName: \"kubernetes.io/projected/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-kube-api-access-7zhlf\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:32:58.679667 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:58.679500    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:32:59.330147 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:59.330108    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx" event={"ID":"f3c063d2-e917-447f-94c7-1ff6cd9f1e7e","Type":"ContainerDied","Data":"e5bc4adc4e90a6472ba156d713a1ce09c81428af7b603db5b790fbf0ab7caa2b"}
Oct 30 01:32:59.330147 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:59.330133    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx"
Oct 30 01:32:59.330375 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:32:59.330171    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5bc4adc4e90a6472ba156d713a1ce09c81428af7b603db5b790fbf0ab7caa2b"
Oct 30 01:33:07.215019 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:33:07.214983    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:33:12.746683 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:33:12.746644    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:33:14.028126 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:33:14.028091    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc"]
Oct 30 01:33:14.039879 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:33:14.039847    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4lbdc"]
Oct 30 01:33:14.259565 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:33:14.259524    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98f86982-37a2-45e5-8028-2bbc099ca613" path="/var/lib/kubelet/pods/98f86982-37a2-45e5-8028-2bbc099ca613/volumes"
Oct 30 01:33:15.852570 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:33:15.852531    6099 scope.go:122] "RemoveContainer" containerID="e8b619f56bf3ec520ac61093ce89573278ce6b146e46c1c10597ed1d917c6688"
Oct 30 01:33:16.629733 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:33:16.629675    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:33:26.028304 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:33:26.028268    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx"]
Oct 30 01:33:26.040227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:33:26.040198    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x9phx"]
Oct 30 01:33:26.260534 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:33:26.260505    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc13f7e2-4f96-4993-8186-a287f49c4886" path="/var/lib/kubelet/pods/dc13f7e2-4f96-4993-8186-a287f49c4886/volumes"
Oct 30 01:33:46.629987 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:33:46.629951    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:33:53.026901 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:33:53.026856    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw"]
Oct 30 01:33:53.038442 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:33:53.038420    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zjvbw"]
Oct 30 01:33:54.260170 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:33:54.260091    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea2543f-eb17-4c3b-a555-c6d8f866834a" path="/var/lib/kubelet/pods/3ea2543f-eb17-4c3b-a555-c6d8f866834a/volumes"
Oct 30 01:34:04.033763 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:04.033724    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg"]
Oct 30 01:34:04.045404 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:04.045374    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7v9fg"]
Oct 30 01:34:04.259551 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:04.259509    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f7f028-efa6-46f1-a697-ae84ea79b026" path="/var/lib/kubelet/pods/f3f7f028-efa6-46f1-a697-ae84ea79b026/volumes"
Oct 30 01:34:15.890274 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:15.890242    6099 scope.go:122] "RemoveContainer" containerID="458e51e5a0c4796dcf449754600464105287d9b7b6d864b5b1bc189f92158ce9"
Oct 30 01:34:15.899677 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:15.899655    6099 scope.go:122] "RemoveContainer" containerID="42c3c703dd4cf083a2a1392026bd7433cbcf77de041338037f3f91a618e124f0"
Oct 30 01:34:16.628523 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:34:16.628486    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:34:26.028023 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:26.027989    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6"]
Oct 30 01:34:26.039778 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:26.039751    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4klp6"]
Oct 30 01:34:26.260402 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:26.260374    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd89c92-dad3-4b87-9a44-7843ecd7b38c" path="/var/lib/kubelet/pods/cdd89c92-dad3-4b87-9a44-7843ecd7b38c/volumes"
Oct 30 01:34:40.363116 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:40.363083    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:34:44.332559 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.332522    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5"]
Oct 30 01:34:44.333191 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.333168    6099 cpu_manager.go:414] "containerMap: removing container" podUID="f3c063d2-e917-447f-94c7-1ff6cd9f1e7e" containerName="revoke-keys"
Oct 30 01:34:44.333287 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.333192    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c063d2-e917-447f-94c7-1ff6cd9f1e7e" containerName="revoke-keys"
Oct 30 01:34:44.333348 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.333321    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3c063d2-e917-447f-94c7-1ff6cd9f1e7e" containerName="revoke-keys"
Oct 30 01:34:44.336440 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.336419    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5"
Oct 30 01:34:44.365407 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.365369    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:34:44.365531 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.365400    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:34:44.365531 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.365469    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:34:44.365531 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.365478    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:34:44.366933 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.366913    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5"]
Oct 30 01:34:44.476206 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.476178    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb57fdac-79f5-42e4-9722-6840fed8ccdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-fspp5\" (UID: \"eb57fdac-79f5-42e4-9722-6840fed8ccdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5"
Oct 30 01:34:44.476342 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.476215    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2cpm\" (UniqueName: \"kubernetes.io/projected/eb57fdac-79f5-42e4-9722-6840fed8ccdc-kube-api-access-m2cpm\") pod \"maas-api-revoke-sub-d576e68b-fspp5\" (UID: \"eb57fdac-79f5-42e4-9722-6840fed8ccdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5"
Oct 30 01:34:44.476342 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.476331    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb57fdac-79f5-42e4-9722-6840fed8ccdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-fspp5\" (UID: \"eb57fdac-79f5-42e4-9722-6840fed8ccdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5"
Oct 30 01:34:44.577636 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.577606    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb57fdac-79f5-42e4-9722-6840fed8ccdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-fspp5\" (UID: \"eb57fdac-79f5-42e4-9722-6840fed8ccdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5"
Oct 30 01:34:44.577800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.577643    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2cpm\" (UniqueName: \"kubernetes.io/projected/eb57fdac-79f5-42e4-9722-6840fed8ccdc-kube-api-access-m2cpm\") pod \"maas-api-revoke-sub-d576e68b-fspp5\" (UID: \"eb57fdac-79f5-42e4-9722-6840fed8ccdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5"
Oct 30 01:34:44.577800 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.577728    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb57fdac-79f5-42e4-9722-6840fed8ccdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-fspp5\" (UID: \"eb57fdac-79f5-42e4-9722-6840fed8ccdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5"
Oct 30 01:34:44.578437 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.578416    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb57fdac-79f5-42e4-9722-6840fed8ccdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-fspp5\" (UID: \"eb57fdac-79f5-42e4-9722-6840fed8ccdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5"
Oct 30 01:34:44.580000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.579980    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb57fdac-79f5-42e4-9722-6840fed8ccdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-fspp5\" (UID: \"eb57fdac-79f5-42e4-9722-6840fed8ccdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5"
Oct 30 01:34:44.600475 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.600415    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2cpm\" (UniqueName: \"kubernetes.io/projected/eb57fdac-79f5-42e4-9722-6840fed8ccdc-kube-api-access-m2cpm\") pod \"maas-api-revoke-sub-d576e68b-fspp5\" (UID: \"eb57fdac-79f5-42e4-9722-6840fed8ccdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5"
Oct 30 01:34:44.649568 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.649541    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5"
Oct 30 01:34:44.946797 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:44.946773    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5"]
Oct 30 01:34:44.949471 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:34:44.949443    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb57fdac_79f5_42e4_9722_6840fed8ccdc.slice/crio-dfb8ca85a7b2df21bafd4986bdeb5b7fbede4d9134fa158eae2a6f9c9050c8d3 WatchSource:0}: Error finding container dfb8ca85a7b2df21bafd4986bdeb5b7fbede4d9134fa158eae2a6f9c9050c8d3: Status 404 returned error can't find the container with id dfb8ca85a7b2df21bafd4986bdeb5b7fbede4d9134fa158eae2a6f9c9050c8d3
Oct 30 01:34:45.963119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:45.963023    6099 generic.go:356] "Generic (PLEG): container finished" podID="eb57fdac-79f5-42e4-9722-6840fed8ccdc" containerID="dfa8e26bf7237c3c98dfb59baf2d165640e7f66ba54eda363f2b6dd85b5655c8" exitCode=0
Oct 30 01:34:45.963119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:45.963077    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5" event={"ID":"eb57fdac-79f5-42e4-9722-6840fed8ccdc","Type":"ContainerDied","Data":"dfa8e26bf7237c3c98dfb59baf2d165640e7f66ba54eda363f2b6dd85b5655c8"}
Oct 30 01:34:45.963525 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:45.963154    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5" event={"ID":"eb57fdac-79f5-42e4-9722-6840fed8ccdc","Type":"ContainerStarted","Data":"dfb8ca85a7b2df21bafd4986bdeb5b7fbede4d9134fa158eae2a6f9c9050c8d3"}
Oct 30 01:34:46.629225 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:34:46.629184    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:34:47.031194 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:47.031167    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td"]
Oct 30 01:34:47.042679 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:47.042655    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jm5td"]
Oct 30 01:34:47.101832 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:47.101809    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5"
Oct 30 01:34:47.203038 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:47.203002    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb57fdac-79f5-42e4-9722-6840fed8ccdc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb57fdac-79f5-42e4-9722-6840fed8ccdc-maas-api-openshift-ca\") pod \"eb57fdac-79f5-42e4-9722-6840fed8ccdc\" (UID: \"eb57fdac-79f5-42e4-9722-6840fed8ccdc\") "
Oct 30 01:34:47.203244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:47.203140    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb57fdac-79f5-42e4-9722-6840fed8ccdc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb57fdac-79f5-42e4-9722-6840fed8ccdc-maas-api-xks-ca\") pod \"eb57fdac-79f5-42e4-9722-6840fed8ccdc\" (UID: \"eb57fdac-79f5-42e4-9722-6840fed8ccdc\") "
Oct 30 01:34:47.203244 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:47.203210    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb57fdac-79f5-42e4-9722-6840fed8ccdc-kube-api-access-m2cpm\" (UniqueName: \"kubernetes.io/projected/eb57fdac-79f5-42e4-9722-6840fed8ccdc-kube-api-access-m2cpm\") pod \"eb57fdac-79f5-42e4-9722-6840fed8ccdc\" (UID: \"eb57fdac-79f5-42e4-9722-6840fed8ccdc\") "
Oct 30 01:34:47.203393 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:47.203368    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb57fdac-79f5-42e4-9722-6840fed8ccdc-maas-api-openshift-ca" pod "eb57fdac-79f5-42e4-9722-6840fed8ccdc" (UID: "eb57fdac-79f5-42e4-9722-6840fed8ccdc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:47.203532 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:47.203514    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb57fdac-79f5-42e4-9722-6840fed8ccdc-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:34:47.205181 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:47.205161    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb57fdac-79f5-42e4-9722-6840fed8ccdc-maas-api-xks-ca" pod "eb57fdac-79f5-42e4-9722-6840fed8ccdc" (UID: "eb57fdac-79f5-42e4-9722-6840fed8ccdc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:34:47.205314 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:47.205293    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb57fdac-79f5-42e4-9722-6840fed8ccdc-kube-api-access-m2cpm" pod "eb57fdac-79f5-42e4-9722-6840fed8ccdc" (UID: "eb57fdac-79f5-42e4-9722-6840fed8ccdc"). InnerVolumeSpecName "kube-api-access-m2cpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:47.304749 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:47.304699    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb57fdac-79f5-42e4-9722-6840fed8ccdc-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:34:47.304749 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:47.304749    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2cpm\" (UniqueName: \"kubernetes.io/projected/eb57fdac-79f5-42e4-9722-6840fed8ccdc-kube-api-access-m2cpm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:34:47.975282 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:47.975256    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5"
Oct 30 01:34:47.975457 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:47.975295    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5" event={"ID":"eb57fdac-79f5-42e4-9722-6840fed8ccdc","Type":"ContainerDied","Data":"dfb8ca85a7b2df21bafd4986bdeb5b7fbede4d9134fa158eae2a6f9c9050c8d3"}
Oct 30 01:34:47.975457 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:47.975342    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfb8ca85a7b2df21bafd4986bdeb5b7fbede4d9134fa158eae2a6f9c9050c8d3"
Oct 30 01:34:48.260308 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:48.260236    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9eec167-9b07-42d0-b70d-658e5b006009" path="/var/lib/kubelet/pods/a9eec167-9b07-42d0-b70d-658e5b006009/volumes"
Oct 30 01:34:55.552011 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:55.551960    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:34:59.315914 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.314976    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz"]
Oct 30 01:34:59.339575 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.315630    6099 cpu_manager.go:414] "containerMap: removing container" podUID="eb57fdac-79f5-42e4-9722-6840fed8ccdc" containerName="revoke-keys"
Oct 30 01:34:59.339857 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.339835    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb57fdac-79f5-42e4-9722-6840fed8ccdc" containerName="revoke-keys"
Oct 30 01:34:59.340242 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.340224    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb57fdac-79f5-42e4-9722-6840fed8ccdc" containerName="revoke-keys"
Oct 30 01:34:59.350968 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.350938    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz"]
Oct 30 01:34:59.351256 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.351237    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz"
Oct 30 01:34:59.377734 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.369603    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:34:59.382000 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.381973    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:34:59.388733 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.382231    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:34:59.388886 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.385089    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:34:59.447155 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.445142    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9406fe-652e-4378-99a9-310525a6ad78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-582nz\" (UID: \"de9406fe-652e-4378-99a9-310525a6ad78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz"
Oct 30 01:34:59.451752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.451002    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9406fe-652e-4378-99a9-310525a6ad78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-582nz\" (UID: \"de9406fe-652e-4378-99a9-310525a6ad78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz"
Oct 30 01:34:59.451752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.451189    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8h6w\" (UniqueName: \"kubernetes.io/projected/de9406fe-652e-4378-99a9-310525a6ad78-kube-api-access-r8h6w\") pod \"maas-api-revoke-sub-d2bc93bc-582nz\" (UID: \"de9406fe-652e-4378-99a9-310525a6ad78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz"
Oct 30 01:34:59.553497 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.553444    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8h6w\" (UniqueName: \"kubernetes.io/projected/de9406fe-652e-4378-99a9-310525a6ad78-kube-api-access-r8h6w\") pod \"maas-api-revoke-sub-d2bc93bc-582nz\" (UID: \"de9406fe-652e-4378-99a9-310525a6ad78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz"
Oct 30 01:34:59.553729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.553585    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9406fe-652e-4378-99a9-310525a6ad78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-582nz\" (UID: \"de9406fe-652e-4378-99a9-310525a6ad78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz"
Oct 30 01:34:59.553729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.553680    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9406fe-652e-4378-99a9-310525a6ad78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-582nz\" (UID: \"de9406fe-652e-4378-99a9-310525a6ad78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz"
Oct 30 01:34:59.554317 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.554284    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9406fe-652e-4378-99a9-310525a6ad78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-582nz\" (UID: \"de9406fe-652e-4378-99a9-310525a6ad78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz"
Oct 30 01:34:59.556104 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.556078    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9406fe-652e-4378-99a9-310525a6ad78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-582nz\" (UID: \"de9406fe-652e-4378-99a9-310525a6ad78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz"
Oct 30 01:34:59.585486 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.585388    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8h6w\" (UniqueName: \"kubernetes.io/projected/de9406fe-652e-4378-99a9-310525a6ad78-kube-api-access-r8h6w\") pod \"maas-api-revoke-sub-d2bc93bc-582nz\" (UID: \"de9406fe-652e-4378-99a9-310525a6ad78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz"
Oct 30 01:34:59.689973 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:34:59.689933    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz"
Oct 30 01:35:00.149633 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:00.149594    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz"]
Oct 30 01:35:00.154503 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:35:00.154474    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde9406fe_652e_4378_99a9_310525a6ad78.slice/crio-48f396d8b7be3c5bd5d528ed8213cf56a17ecb4a24989e94f30a6ead161e2850 WatchSource:0}: Error finding container 48f396d8b7be3c5bd5d528ed8213cf56a17ecb4a24989e94f30a6ead161e2850: Status 404 returned error can't find the container with id 48f396d8b7be3c5bd5d528ed8213cf56a17ecb4a24989e94f30a6ead161e2850
Oct 30 01:35:01.061657 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:01.061617    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz" event={"ID":"de9406fe-652e-4378-99a9-310525a6ad78","Type":"ContainerStarted","Data":"2c9c9fc4f8b27890b331b115a71156f4b85be43cb00ef96e8e000f5b80b93a61"}
Oct 30 01:35:01.062072 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:01.061676    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz" event={"ID":"de9406fe-652e-4378-99a9-310525a6ad78","Type":"ContainerStarted","Data":"48f396d8b7be3c5bd5d528ed8213cf56a17ecb4a24989e94f30a6ead161e2850"}
Oct 30 01:35:01.114106 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:01.114051    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz" podStartSLOduration=2.114038074 podStartE2EDuration="2.114038074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:01.112258416 +0000 UTC m=+3594.969166172" watchObservedRunningTime="2026-10-30 01:35:01.114038074 +0000 UTC m=+3594.970945870"
Oct 30 01:35:02.069312 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:02.069280    6099 generic.go:356] "Generic (PLEG): container finished" podID="de9406fe-652e-4378-99a9-310525a6ad78" containerID="2c9c9fc4f8b27890b331b115a71156f4b85be43cb00ef96e8e000f5b80b93a61" exitCode=0
Oct 30 01:35:02.069770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:02.069373    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz" event={"ID":"de9406fe-652e-4378-99a9-310525a6ad78","Type":"ContainerDied","Data":"2c9c9fc4f8b27890b331b115a71156f4b85be43cb00ef96e8e000f5b80b93a61"}
Oct 30 01:35:03.212259 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:03.212234    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz"
Oct 30 01:35:03.340272 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:03.340184    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de9406fe-652e-4378-99a9-310525a6ad78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9406fe-652e-4378-99a9-310525a6ad78-maas-api-xks-ca\") pod \"de9406fe-652e-4378-99a9-310525a6ad78\" (UID: \"de9406fe-652e-4378-99a9-310525a6ad78\") "
Oct 30 01:35:03.340453 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:03.340285    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de9406fe-652e-4378-99a9-310525a6ad78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9406fe-652e-4378-99a9-310525a6ad78-maas-api-openshift-ca\") pod \"de9406fe-652e-4378-99a9-310525a6ad78\" (UID: \"de9406fe-652e-4378-99a9-310525a6ad78\") "
Oct 30 01:35:03.340453 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:03.340390    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de9406fe-652e-4378-99a9-310525a6ad78-kube-api-access-r8h6w\" (UniqueName: \"kubernetes.io/projected/de9406fe-652e-4378-99a9-310525a6ad78-kube-api-access-r8h6w\") pod \"de9406fe-652e-4378-99a9-310525a6ad78\" (UID: \"de9406fe-652e-4378-99a9-310525a6ad78\") "
Oct 30 01:35:03.340678 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:03.340635    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de9406fe-652e-4378-99a9-310525a6ad78-maas-api-openshift-ca" pod "de9406fe-652e-4378-99a9-310525a6ad78" (UID: "de9406fe-652e-4378-99a9-310525a6ad78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:35:03.340815 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:03.340796    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9406fe-652e-4378-99a9-310525a6ad78-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:35:03.342456 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:03.342432    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de9406fe-652e-4378-99a9-310525a6ad78-maas-api-xks-ca" pod "de9406fe-652e-4378-99a9-310525a6ad78" (UID: "de9406fe-652e-4378-99a9-310525a6ad78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:35:03.342599 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:03.342576    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de9406fe-652e-4378-99a9-310525a6ad78-kube-api-access-r8h6w" pod "de9406fe-652e-4378-99a9-310525a6ad78" (UID: "de9406fe-652e-4378-99a9-310525a6ad78"). InnerVolumeSpecName "kube-api-access-r8h6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:35:03.442299 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:03.442244    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8h6w\" (UniqueName: \"kubernetes.io/projected/de9406fe-652e-4378-99a9-310525a6ad78-kube-api-access-r8h6w\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:35:03.442299 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:03.442293    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9406fe-652e-4378-99a9-310525a6ad78-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:35:04.081580 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:04.081541    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz" event={"ID":"de9406fe-652e-4378-99a9-310525a6ad78","Type":"ContainerDied","Data":"48f396d8b7be3c5bd5d528ed8213cf56a17ecb4a24989e94f30a6ead161e2850"}
Oct 30 01:35:04.081806 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:04.081606    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48f396d8b7be3c5bd5d528ed8213cf56a17ecb4a24989e94f30a6ead161e2850"
Oct 30 01:35:04.081806 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:04.081556    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz"
Oct 30 01:35:07.009826 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:07.009798    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 01:35:07.024638 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:07.024613    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 01:35:11.463317 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.463273    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"]
Oct 30 01:35:11.463815 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.463799    6099 cpu_manager.go:414] "containerMap: removing container" podUID="de9406fe-652e-4378-99a9-310525a6ad78" containerName="revoke-keys"
Oct 30 01:35:11.463876 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.463817    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de9406fe-652e-4378-99a9-310525a6ad78" containerName="revoke-keys"
Oct 30 01:35:11.463927 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.463896    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="de9406fe-652e-4378-99a9-310525a6ad78" containerName="revoke-keys"
Oct 30 01:35:11.466925 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.466907    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.482834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.482807    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-7e040d-openshift-default-dockercfg-qw6mk\""
Oct 30 01:35:11.496333 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.496305    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"]
Oct 30 01:35:11.623975 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.623937    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55fd1643-3498-4370-a0f8-a8af2204111f-istio-token\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.623975 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.623979    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-credential-socket\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.624250 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.624003    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-workload-certs\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.624250 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.624150    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-istio-envoy\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.624250 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.624226    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55fd1643-3498-4370-a0f8-a8af2204111f-istio-podinfo\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.624411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.624266    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55fd1643-3498-4370-a0f8-a8af2204111f-istiod-ca-cert\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.624411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.624322    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sc8b\" (UniqueName: \"kubernetes.io/projected/55fd1643-3498-4370-a0f8-a8af2204111f-kube-api-access-4sc8b\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.624411 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.624363    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-istio-data\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.624527 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.624406    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-workload-socket\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.725837 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.725754    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55fd1643-3498-4370-a0f8-a8af2204111f-istio-token\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.725837 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.725800    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-credential-socket\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.725837 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.725826    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-workload-certs\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.726180 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.725863    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-istio-envoy\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.726180 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.725908    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55fd1643-3498-4370-a0f8-a8af2204111f-istio-podinfo\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.726180 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.725944    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55fd1643-3498-4370-a0f8-a8af2204111f-istiod-ca-cert\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.726180 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.725981    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sc8b\" (UniqueName: \"kubernetes.io/projected/55fd1643-3498-4370-a0f8-a8af2204111f-kube-api-access-4sc8b\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.726180 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.726012    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-istio-data\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.726180 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.726044    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-workload-socket\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.726531 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.726253    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-credential-socket\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.726531 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.726418    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-workload-certs\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.726531 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.726461    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-workload-socket\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.726691 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.726603    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-istio-data\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.726904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.726884    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55fd1643-3498-4370-a0f8-a8af2204111f-istiod-ca-cert\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.728263 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.728238    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-istio-envoy\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.728432 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.728407    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55fd1643-3498-4370-a0f8-a8af2204111f-istio-podinfo\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.748563 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.748541    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55fd1643-3498-4370-a0f8-a8af2204111f-istio-token\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.752466 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.752446    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sc8b\" (UniqueName: \"kubernetes.io/projected/55fd1643-3498-4370-a0f8-a8af2204111f-kube-api-access-4sc8b\") pod \"e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") " pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:11.779904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:11.779884    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:12.091408 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:12.091384    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"]
Oct 30 01:35:12.093126 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:35:12.093099    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55fd1643_3498_4370_a0f8_a8af2204111f.slice/crio-2b2440967513fa2209958399c48426744c866669bf389c32e3451afa3c7ab24b WatchSource:0}: Error finding container 2b2440967513fa2209958399c48426744c866669bf389c32e3451afa3c7ab24b: Status 404 returned error can't find the container with id 2b2440967513fa2209958399c48426744c866669bf389c32e3451afa3c7ab24b
Oct 30 01:35:12.095132 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:12.095103    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:35:12.095204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:12.095164    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:35:12.095204 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:12.095195    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:35:12.121810 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:12.121785    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq" event={"ID":"55fd1643-3498-4370-a0f8-a8af2204111f","Type":"ContainerStarted","Data":"2b2440967513fa2209958399c48426744c866669bf389c32e3451afa3c7ab24b"}
Oct 30 01:35:13.128227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:13.128184    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq" event={"ID":"55fd1643-3498-4370-a0f8-a8af2204111f","Type":"ContainerStarted","Data":"0cd549c7715b94409609f3a46322852f14bf19d0453a3c38429b380548840350"}
Oct 30 01:35:13.196791 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:13.196737    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq" podStartSLOduration=2.196723479 podStartE2EDuration="2.196723479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:13.184679307 +0000 UTC m=+3607.041587065" watchObservedRunningTime="2026-10-30 01:35:13.196723479 +0000 UTC m=+3607.053631227"
Oct 30 01:35:13.780090 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:13.780052    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:13.785346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:13.785320    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:14.133084 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:14.133053    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:14.134078 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:14.134059    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:15.927430 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:15.927399    6099 scope.go:122] "RemoveContainer" containerID="ddf61b3f80f951b7c76b89f5b910bb26d5a71974e9b30b67cfd04eaa748ebfe2"
Oct 30 01:35:15.936666 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:15.936644    6099 scope.go:122] "RemoveContainer" containerID="1d168228f4e55158b7d60af8c9a3a041ec33a3902e4b8f5b380cc109a8238576"
Oct 30 01:35:15.946452 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:15.946428    6099 scope.go:122] "RemoveContainer" containerID="67a9a16419ef5876e08c90ae3cd70fc6204c8d4b8d27596462dcf6270d0054ac"
Oct 30 01:35:16.628784 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:35:16.628751    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:35:16.904243 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:16.904155    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"]
Oct 30 01:35:16.909406 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:16.909385    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"
Oct 30 01:35:16.926403 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:16.926381    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k5bc5\""
Oct 30 01:35:16.938633 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:16.938597    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:35:16.939235 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:16.938890    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:35:16.941331 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:16.941307    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"]
Oct 30 01:35:16.975959 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:16.975929    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba150208-a008-4eed-b889-a5cf36854373-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj\" (UID: \"ba150208-a008-4eed-b889-a5cf36854373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"
Oct 30 01:35:16.976130 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:16.975968    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba150208-a008-4eed-b889-a5cf36854373-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj\" (UID: \"ba150208-a008-4eed-b889-a5cf36854373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"
Oct 30 01:35:16.976130 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:16.976091    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p86ms\" (UniqueName: \"kubernetes.io/projected/ba150208-a008-4eed-b889-a5cf36854373-kube-api-access-p86ms\") pod \"maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj\" (UID: \"ba150208-a008-4eed-b889-a5cf36854373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"
Oct 30 01:35:16.976208 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:16.976146    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba150208-a008-4eed-b889-a5cf36854373-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj\" (UID: \"ba150208-a008-4eed-b889-a5cf36854373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"
Oct 30 01:35:16.989272 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:16.989244    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-7e040d-serving-cert\""
Oct 30 01:35:16.989400 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:16.989286    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 01:35:17.076937 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.076903    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p86ms\" (UniqueName: \"kubernetes.io/projected/ba150208-a008-4eed-b889-a5cf36854373-kube-api-access-p86ms\") pod \"maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj\" (UID: \"ba150208-a008-4eed-b889-a5cf36854373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"
Oct 30 01:35:17.076937 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.076945    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba150208-a008-4eed-b889-a5cf36854373-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj\" (UID: \"ba150208-a008-4eed-b889-a5cf36854373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"
Oct 30 01:35:17.077224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.077028    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba150208-a008-4eed-b889-a5cf36854373-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj\" (UID: \"ba150208-a008-4eed-b889-a5cf36854373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"
Oct 30 01:35:17.077224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.077051    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba150208-a008-4eed-b889-a5cf36854373-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj\" (UID: \"ba150208-a008-4eed-b889-a5cf36854373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"
Oct 30 01:35:17.077733 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.077678    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba150208-a008-4eed-b889-a5cf36854373-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj\" (UID: \"ba150208-a008-4eed-b889-a5cf36854373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"
Oct 30 01:35:17.079767 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.079739    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba150208-a008-4eed-b889-a5cf36854373-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj\" (UID: \"ba150208-a008-4eed-b889-a5cf36854373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"
Oct 30 01:35:17.079767 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.079757    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba150208-a008-4eed-b889-a5cf36854373-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj\" (UID: \"ba150208-a008-4eed-b889-a5cf36854373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"
Oct 30 01:35:17.100546 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.100515    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p86ms\" (UniqueName: \"kubernetes.io/projected/ba150208-a008-4eed-b889-a5cf36854373-kube-api-access-p86ms\") pod \"maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj\" (UID: \"ba150208-a008-4eed-b889-a5cf36854373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"
Oct 30 01:35:17.223528 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.223446    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"
Oct 30 01:35:17.337960 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.337925    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927"]
Oct 30 01:35:17.342905 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.342882    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927"
Oct 30 01:35:17.364044 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.363849    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-7e040d\""
Oct 30 01:35:17.366012 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.365983    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927"]
Oct 30 01:35:17.379380 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.379354    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fff1cb9a-860c-4b1c-b59a-3246c8184e8d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927\" (UID: \"fff1cb9a-860c-4b1c-b59a-3246c8184e8d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927"
Oct 30 01:35:17.379529 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.379425    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs2vm\" (UniqueName: \"kubernetes.io/projected/fff1cb9a-860c-4b1c-b59a-3246c8184e8d-kube-api-access-rs2vm\") pod \"payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927\" (UID: \"fff1cb9a-860c-4b1c-b59a-3246c8184e8d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927"
Oct 30 01:35:17.487856 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.480521    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fff1cb9a-860c-4b1c-b59a-3246c8184e8d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927\" (UID: \"fff1cb9a-860c-4b1c-b59a-3246c8184e8d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927"
Oct 30 01:35:17.487856 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.480632    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs2vm\" (UniqueName: \"kubernetes.io/projected/fff1cb9a-860c-4b1c-b59a-3246c8184e8d-kube-api-access-rs2vm\") pod \"payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927\" (UID: \"fff1cb9a-860c-4b1c-b59a-3246c8184e8d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927"
Oct 30 01:35:17.487856 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.482512    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fff1cb9a-860c-4b1c-b59a-3246c8184e8d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927\" (UID: \"fff1cb9a-860c-4b1c-b59a-3246c8184e8d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927"
Oct 30 01:35:17.505649 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.505615    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs2vm\" (UniqueName: \"kubernetes.io/projected/fff1cb9a-860c-4b1c-b59a-3246c8184e8d-kube-api-access-rs2vm\") pod \"payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927\" (UID: \"fff1cb9a-860c-4b1c-b59a-3246c8184e8d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927"
Oct 30 01:35:17.538570 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.538545    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"]
Oct 30 01:35:17.540059 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:35:17.540032    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba150208_a008_4eed_b889_a5cf36854373.slice/crio-87ea1c4c071bcff921ead64e037363f892fedda5527db7d114c933576d63207a WatchSource:0}: Error finding container 87ea1c4c071bcff921ead64e037363f892fedda5527db7d114c933576d63207a: Status 404 returned error can't find the container with id 87ea1c4c071bcff921ead64e037363f892fedda5527db7d114c933576d63207a
Oct 30 01:35:17.656491 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.656450    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927"
Oct 30 01:35:17.956322 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:17.956180    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927"]
Oct 30 01:35:17.957402 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:35:17.957380    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff1cb9a_860c_4b1c_b59a_3246c8184e8d.slice/crio-81a122ededccde379da9ca3684f7cdb73d92c519a753de6a763327b2bff0f255 WatchSource:0}: Error finding container 81a122ededccde379da9ca3684f7cdb73d92c519a753de6a763327b2bff0f255: Status 404 returned error can't find the container with id 81a122ededccde379da9ca3684f7cdb73d92c519a753de6a763327b2bff0f255
Oct 30 01:35:18.163496 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:18.163356    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927" event={"ID":"fff1cb9a-860c-4b1c-b59a-3246c8184e8d","Type":"ContainerStarted","Data":"7e2787205ca816db8d8ed02333723bbf8b2170c19c800c416c998b215bfa3da2"}
Oct 30 01:35:18.163496 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:18.163428    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927" event={"ID":"fff1cb9a-860c-4b1c-b59a-3246c8184e8d","Type":"ContainerStarted","Data":"81a122ededccde379da9ca3684f7cdb73d92c519a753de6a763327b2bff0f255"}
Oct 30 01:35:18.163868 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:18.163646    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927"
Oct 30 01:35:18.166364 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:18.166333    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj" event={"ID":"ba150208-a008-4eed-b889-a5cf36854373","Type":"ContainerStarted","Data":"974710715d8003631fab797563c891f787a90791b8c7d9b35798e976fda3c994"}
Oct 30 01:35:18.166517 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:18.166388    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj" event={"ID":"ba150208-a008-4eed-b889-a5cf36854373","Type":"ContainerStarted","Data":"87ea1c4c071bcff921ead64e037363f892fedda5527db7d114c933576d63207a"}
Oct 30 01:35:18.166517 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:18.166462    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"
Oct 30 01:35:18.203008 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:18.202952    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927" podStartSLOduration=1.20293479 podStartE2EDuration="1.20293479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:18.202202904 +0000 UTC m=+3612.059110660" watchObservedRunningTime="2026-10-30 01:35:18.20293479 +0000 UTC m=+3612.059842546"
Oct 30 01:35:18.249989 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:18.249932    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj" podStartSLOduration=1.887369743 podStartE2EDuration="2.249914861s" totalImagesPullingTime="362.545118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:18.248072926 +0000 UTC m=+3612.104980683" watchObservedRunningTime="2026-10-30 01:35:18.249914861 +0000 UTC m=+3612.106822630"
Oct 30 01:35:20.575903 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:20.575841    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:35:23.029049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:23.029006    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx"]
Oct 30 01:35:23.040806 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:23.040771    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-trzfx"]
Oct 30 01:35:24.192910 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:24.192879    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"
Oct 30 01:35:24.260228 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:24.260198    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b2039c6-8ff3-42a9-bc9d-b5e85cb76177" path="/var/lib/kubelet/pods/2b2039c6-8ff3-42a9-bc9d-b5e85cb76177/volumes"
Oct 30 01:35:29.174151 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:29.174098    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927"
Oct 30 01:35:29.764369 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:29.764120    6099 patch_prober.go:29] interesting pod/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.99:15021/healthz/ready\": read tcp 10.134.0.2:39396->10.134.1.99:15021: read: connection reset by peer" start-of-body=
Oct 30 01:35:29.764369 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:29.764277    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq" podUID="55fd1643-3498-4370-a0f8-a8af2204111f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.99:15021/healthz/ready\": read tcp 10.134.0.2:39396->10.134.1.99:15021: read: connection reset by peer"
Oct 30 01:35:30.245845 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:30.245808    6099 generic.go:356] "Generic (PLEG): container finished" podID="55fd1643-3498-4370-a0f8-a8af2204111f" containerID="0cd549c7715b94409609f3a46322852f14bf19d0453a3c38429b380548840350" exitCode=0
Oct 30 01:35:30.246373 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:30.245862    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq" event={"ID":"55fd1643-3498-4370-a0f8-a8af2204111f","Type":"ContainerDied","Data":"0cd549c7715b94409609f3a46322852f14bf19d0453a3c38429b380548840350"}
Oct 30 01:35:30.246373 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:30.246232    6099 scope.go:122] "RemoveContainer" containerID="0cd549c7715b94409609f3a46322852f14bf19d0453a3c38429b380548840350"
Oct 30 01:35:30.247421 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:30.247389    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:35:30.247534 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:30.247455    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:35:30.247534 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:30.247485    6099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:35:31.253433 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:31.253395    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq" event={"ID":"55fd1643-3498-4370-a0f8-a8af2204111f","Type":"ContainerStarted","Data":"9c54f801b2af9a5abcf458a8cca838fffe503bc9e3e03e50bf8e4fc90c9819d8"}
Oct 30 01:35:31.782108 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:31.782072    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:31.782482 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:31.782457    6099 patch_prober.go:29] interesting pod/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused" start-of-body=
Oct 30 01:35:31.782668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:31.782641    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq" podUID="55fd1643-3498-4370-a0f8-a8af2204111f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused"
Oct 30 01:35:32.785167 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:32.785128    6099 patch_prober.go:29] interesting pod/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused" start-of-body=
Oct 30 01:35:32.789860 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:32.785726    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq" podUID="55fd1643-3498-4370-a0f8-a8af2204111f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused"
Oct 30 01:35:33.785137 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:33.785094    6099 patch_prober.go:29] interesting pod/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused" start-of-body=
Oct 30 01:35:33.785452 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:33.785421    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq" podUID="55fd1643-3498-4370-a0f8-a8af2204111f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused"
Oct 30 01:35:34.789102 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:34.789065    6099 patch_prober.go:29] interesting pod/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused" start-of-body=
Oct 30 01:35:34.789674 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:34.789636    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq" podUID="55fd1643-3498-4370-a0f8-a8af2204111f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused"
Oct 30 01:35:35.046861 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:35.046784    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:35:35.784119 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:35.784082    6099 patch_prober.go:29] interesting pod/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused" start-of-body=
Oct 30 01:35:35.784448 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:35.784406    6099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq" podUID="55fd1643-3498-4370-a0f8-a8af2204111f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused"
Oct 30 01:35:36.785303 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:36.785267    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:36.785861 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:36.785739    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:36.787201 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:36.787176    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:35:46.629005 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:35:46.628963    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:35:50.141939 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:35:50.141906    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:36:03.564325 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:03.564291    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:36:13.514239 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:13.514202    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927"]
Oct 30 01:36:13.737232 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:13.737199    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"]
Oct 30 01:36:13.737497 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:13.737449    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj" podUID="ba150208-a008-4eed-b889-a5cf36854373" containerName="maas-api" containerID="cri-o://974710715d8003631fab797563c891f787a90791b8c7d9b35798e976fda3c994" gracePeriod=30
Oct 30 01:36:14.205499 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.205470    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"
Oct 30 01:36:14.305661 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.305632    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba150208-a008-4eed-b889-a5cf36854373-kube-api-access-p86ms\" (UniqueName: \"kubernetes.io/projected/ba150208-a008-4eed-b889-a5cf36854373-kube-api-access-p86ms\") pod \"ba150208-a008-4eed-b889-a5cf36854373\" (UID: \"ba150208-a008-4eed-b889-a5cf36854373\") "
Oct 30 01:36:14.305831 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.305698    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba150208-a008-4eed-b889-a5cf36854373-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba150208-a008-4eed-b889-a5cf36854373-openshift-service-ca\") pod \"ba150208-a008-4eed-b889-a5cf36854373\" (UID: \"ba150208-a008-4eed-b889-a5cf36854373\") "
Oct 30 01:36:14.305831 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.305771    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba150208-a008-4eed-b889-a5cf36854373-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba150208-a008-4eed-b889-a5cf36854373-maas-api-tls\") pod \"ba150208-a008-4eed-b889-a5cf36854373\" (UID: \"ba150208-a008-4eed-b889-a5cf36854373\") "
Oct 30 01:36:14.305831 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.305795    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba150208-a008-4eed-b889-a5cf36854373-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba150208-a008-4eed-b889-a5cf36854373-metrics-certs\") pod \"ba150208-a008-4eed-b889-a5cf36854373\" (UID: \"ba150208-a008-4eed-b889-a5cf36854373\") "
Oct 30 01:36:14.306142 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.306109    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba150208-a008-4eed-b889-a5cf36854373-openshift-service-ca" pod "ba150208-a008-4eed-b889-a5cf36854373" (UID: "ba150208-a008-4eed-b889-a5cf36854373"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:36:14.307924 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.307899    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba150208-a008-4eed-b889-a5cf36854373-maas-api-tls" pod "ba150208-a008-4eed-b889-a5cf36854373" (UID: "ba150208-a008-4eed-b889-a5cf36854373"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:36:14.308005 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.307976    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba150208-a008-4eed-b889-a5cf36854373-kube-api-access-p86ms" pod "ba150208-a008-4eed-b889-a5cf36854373" (UID: "ba150208-a008-4eed-b889-a5cf36854373"). InnerVolumeSpecName "kube-api-access-p86ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:14.308084 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.308067    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba150208-a008-4eed-b889-a5cf36854373-metrics-certs" pod "ba150208-a008-4eed-b889-a5cf36854373" (UID: "ba150208-a008-4eed-b889-a5cf36854373"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:36:14.407466 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.407431    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba150208-a008-4eed-b889-a5cf36854373-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:36:14.407466 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.407462    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba150208-a008-4eed-b889-a5cf36854373-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:36:14.407655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.407475    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba150208-a008-4eed-b889-a5cf36854373-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:36:14.407655 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.407488    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p86ms\" (UniqueName: \"kubernetes.io/projected/ba150208-a008-4eed-b889-a5cf36854373-kube-api-access-p86ms\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:36:14.517934 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.517903    6099 generic.go:356] "Generic (PLEG): container finished" podID="ba150208-a008-4eed-b889-a5cf36854373" containerID="974710715d8003631fab797563c891f787a90791b8c7d9b35798e976fda3c994" exitCode=0
Oct 30 01:36:14.518320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.517953    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"
Oct 30 01:36:14.518320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.517984    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj" event={"ID":"ba150208-a008-4eed-b889-a5cf36854373","Type":"ContainerDied","Data":"974710715d8003631fab797563c891f787a90791b8c7d9b35798e976fda3c994"}
Oct 30 01:36:14.518320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.518033    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj" event={"ID":"ba150208-a008-4eed-b889-a5cf36854373","Type":"ContainerDied","Data":"87ea1c4c071bcff921ead64e037363f892fedda5527db7d114c933576d63207a"}
Oct 30 01:36:14.518320 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.518055    6099 scope.go:122] "RemoveContainer" containerID="974710715d8003631fab797563c891f787a90791b8c7d9b35798e976fda3c994"
Oct 30 01:36:14.528242 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.528225    6099 scope.go:122] "RemoveContainer" containerID="974710715d8003631fab797563c891f787a90791b8c7d9b35798e976fda3c994"
Oct 30 01:36:14.528505 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:36:14.528484    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"974710715d8003631fab797563c891f787a90791b8c7d9b35798e976fda3c994\": container with ID starting with 974710715d8003631fab797563c891f787a90791b8c7d9b35798e976fda3c994 not found: ID does not exist" containerID="974710715d8003631fab797563c891f787a90791b8c7d9b35798e976fda3c994"
Oct 30 01:36:14.528575 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.528512    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974710715d8003631fab797563c891f787a90791b8c7d9b35798e976fda3c994"} err="failed to get container status \"974710715d8003631fab797563c891f787a90791b8c7d9b35798e976fda3c994\": rpc error: code = NotFound desc = could not find container \"974710715d8003631fab797563c891f787a90791b8c7d9b35798e976fda3c994\": container with ID starting with 974710715d8003631fab797563c891f787a90791b8c7d9b35798e976fda3c994 not found: ID does not exist"
Oct 30 01:36:14.568455 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.568396    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"]
Oct 30 01:36:14.580240 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:14.580218    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj"]
Oct 30 01:36:15.977092 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:15.977062    6099 scope.go:122] "RemoveContainer" containerID="2e55d6fe9adf6c6aa881790e41af17ddc943ee0d6d4a31a8e13f360b93fded6b"
Oct 30 01:36:15.985832 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:15.985810    6099 scope.go:122] "RemoveContainer" containerID="723eb9d772e36b5f3b336e1f5dbe63ec21031b11f12f4b78a824e765b5121f7b"
Oct 30 01:36:16.173850 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:16.173809    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:36:16.260571 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:16.260494    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba150208-a008-4eed-b889-a5cf36854373" path="/var/lib/kubelet/pods/ba150208-a008-4eed-b889-a5cf36854373/volumes"
Oct 30 01:36:16.629748 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:36:16.629685    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:36:17.172750 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:17.172694    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7e040d-75c748cd69-pv5sj" podUID="ba150208-a008-4eed-b889-a5cf36854373" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.1.100:8443/health\": context deadline exceeded"
Oct 30 01:36:18.514767 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:18.514727    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927" podUID="fff1cb9a-860c-4b1c-b59a-3246c8184e8d" containerName="payload-processing" containerID="cri-o://7e2787205ca816db8d8ed02333723bbf8b2170c19c800c416c998b215bfa3da2" gracePeriod=25
Oct 30 01:36:19.173572 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:19.173537    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927" podUID="fff1cb9a-860c-4b1c-b59a-3246c8184e8d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:36:21.673220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:21.670409    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"]
Oct 30 01:36:21.673220 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:21.670652    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq" podUID="55fd1643-3498-4370-a0f8-a8af2204111f" containerName="istio-proxy" containerID="cri-o://9c54f801b2af9a5abcf458a8cca838fffe503bc9e3e03e50bf8e4fc90c9819d8" gracePeriod=30
Oct 30 01:36:21.788127 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:21.788092    6099 patch_prober.go:29] interesting pod/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:36:21.788299 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:21.788161    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq" podUID="55fd1643-3498-4370-a0f8-a8af2204111f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:36:26.942905 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:26.942839    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:36:27.032858 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.032825    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55fd1643-3498-4370-a0f8-a8af2204111f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55fd1643-3498-4370-a0f8-a8af2204111f-istiod-ca-cert\") pod \"55fd1643-3498-4370-a0f8-a8af2204111f\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") "
Oct 30 01:36:27.033066 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.032885    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-istio-data\") pod \"55fd1643-3498-4370-a0f8-a8af2204111f\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") "
Oct 30 01:36:27.033066 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.032941    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55fd1643-3498-4370-a0f8-a8af2204111f-istio-token\" (UniqueName: \"kubernetes.io/projected/55fd1643-3498-4370-a0f8-a8af2204111f-istio-token\") pod \"55fd1643-3498-4370-a0f8-a8af2204111f\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") "
Oct 30 01:36:27.033066 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.032989    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-workload-certs\") pod \"55fd1643-3498-4370-a0f8-a8af2204111f\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") "
Oct 30 01:36:27.033066 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.033058    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-istio-envoy\") pod \"55fd1643-3498-4370-a0f8-a8af2204111f\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") "
Oct 30 01:36:27.033305 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.033091    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55fd1643-3498-4370-a0f8-a8af2204111f-kube-api-access-4sc8b\" (UniqueName: \"kubernetes.io/projected/55fd1643-3498-4370-a0f8-a8af2204111f-kube-api-access-4sc8b\") pod \"55fd1643-3498-4370-a0f8-a8af2204111f\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") "
Oct 30 01:36:27.033305 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.033130    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55fd1643-3498-4370-a0f8-a8af2204111f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55fd1643-3498-4370-a0f8-a8af2204111f-istio-podinfo\") pod \"55fd1643-3498-4370-a0f8-a8af2204111f\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") "
Oct 30 01:36:27.033305 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.033217    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-workload-socket\") pod \"55fd1643-3498-4370-a0f8-a8af2204111f\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") "
Oct 30 01:36:27.033305 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.033264    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-istio-data" pod "55fd1643-3498-4370-a0f8-a8af2204111f" (UID: "55fd1643-3498-4370-a0f8-a8af2204111f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:36:27.034070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.033306    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-credential-socket\") pod \"55fd1643-3498-4370-a0f8-a8af2204111f\" (UID: \"55fd1643-3498-4370-a0f8-a8af2204111f\") "
Oct 30 01:36:27.034070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.033317    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55fd1643-3498-4370-a0f8-a8af2204111f-istiod-ca-cert" pod "55fd1643-3498-4370-a0f8-a8af2204111f" (UID: "55fd1643-3498-4370-a0f8-a8af2204111f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:36:27.034070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.033616    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-workload-socket" pod "55fd1643-3498-4370-a0f8-a8af2204111f" (UID: "55fd1643-3498-4370-a0f8-a8af2204111f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:36:27.034070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.033676    6099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-istio-data\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:36:27.034070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.033696    6099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55fd1643-3498-4370-a0f8-a8af2204111f-istiod-ca-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:36:27.034070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.033777    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-credential-socket" pod "55fd1643-3498-4370-a0f8-a8af2204111f" (UID: "55fd1643-3498-4370-a0f8-a8af2204111f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:36:27.034070 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.033848    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-workload-certs" pod "55fd1643-3498-4370-a0f8-a8af2204111f" (UID: "55fd1643-3498-4370-a0f8-a8af2204111f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:36:27.035785 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.035739    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-istio-envoy" pod "55fd1643-3498-4370-a0f8-a8af2204111f" (UID: "55fd1643-3498-4370-a0f8-a8af2204111f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:36:27.035933 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.035868    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55fd1643-3498-4370-a0f8-a8af2204111f-kube-api-access-4sc8b" pod "55fd1643-3498-4370-a0f8-a8af2204111f" (UID: "55fd1643-3498-4370-a0f8-a8af2204111f"). InnerVolumeSpecName "kube-api-access-4sc8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:27.036281 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.036256    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55fd1643-3498-4370-a0f8-a8af2204111f-istio-podinfo" pod "55fd1643-3498-4370-a0f8-a8af2204111f" (UID: "55fd1643-3498-4370-a0f8-a8af2204111f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:36:27.036392 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.036311    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55fd1643-3498-4370-a0f8-a8af2204111f-istio-token" pod "55fd1643-3498-4370-a0f8-a8af2204111f" (UID: "55fd1643-3498-4370-a0f8-a8af2204111f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:27.134949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.134913    6099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-istio-envoy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:36:27.134949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.134941    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sc8b\" (UniqueName: \"kubernetes.io/projected/55fd1643-3498-4370-a0f8-a8af2204111f-kube-api-access-4sc8b\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:36:27.134949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.134951    6099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55fd1643-3498-4370-a0f8-a8af2204111f-istio-podinfo\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:36:27.135229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.134962    6099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-workload-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:36:27.135229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.134975    6099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-credential-socket\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:36:27.135229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.134988    6099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55fd1643-3498-4370-a0f8-a8af2204111f-istio-token\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:36:27.135229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.134996    6099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55fd1643-3498-4370-a0f8-a8af2204111f-workload-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:36:27.586116 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.586071    6099 generic.go:356] "Generic (PLEG): container finished" podID="55fd1643-3498-4370-a0f8-a8af2204111f" containerID="9c54f801b2af9a5abcf458a8cca838fffe503bc9e3e03e50bf8e4fc90c9819d8" exitCode=0
Oct 30 01:36:27.586294 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.586116    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq" event={"ID":"55fd1643-3498-4370-a0f8-a8af2204111f","Type":"ContainerDied","Data":"9c54f801b2af9a5abcf458a8cca838fffe503bc9e3e03e50bf8e4fc90c9819d8"}
Oct 30 01:36:27.586294 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.586135    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"
Oct 30 01:36:27.586294 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.586171    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq" event={"ID":"55fd1643-3498-4370-a0f8-a8af2204111f","Type":"ContainerDied","Data":"2b2440967513fa2209958399c48426744c866669bf389c32e3451afa3c7ab24b"}
Oct 30 01:36:27.586294 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.586194    6099 scope.go:122] "RemoveContainer" containerID="9c54f801b2af9a5abcf458a8cca838fffe503bc9e3e03e50bf8e4fc90c9819d8"
Oct 30 01:36:27.600294 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.598963    6099 scope.go:122] "RemoveContainer" containerID="0cd549c7715b94409609f3a46322852f14bf19d0453a3c38429b380548840350"
Oct 30 01:36:27.609473 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.609388    6099 scope.go:122] "RemoveContainer" containerID="9c54f801b2af9a5abcf458a8cca838fffe503bc9e3e03e50bf8e4fc90c9819d8"
Oct 30 01:36:27.609689 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:36:27.609669    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c54f801b2af9a5abcf458a8cca838fffe503bc9e3e03e50bf8e4fc90c9819d8\": container with ID starting with 9c54f801b2af9a5abcf458a8cca838fffe503bc9e3e03e50bf8e4fc90c9819d8 not found: ID does not exist" containerID="9c54f801b2af9a5abcf458a8cca838fffe503bc9e3e03e50bf8e4fc90c9819d8"
Oct 30 01:36:27.609810 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.609726    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c54f801b2af9a5abcf458a8cca838fffe503bc9e3e03e50bf8e4fc90c9819d8"} err="failed to get container status \"9c54f801b2af9a5abcf458a8cca838fffe503bc9e3e03e50bf8e4fc90c9819d8\": rpc error: code = NotFound desc = could not find container \"9c54f801b2af9a5abcf458a8cca838fffe503bc9e3e03e50bf8e4fc90c9819d8\": container with ID starting with 9c54f801b2af9a5abcf458a8cca838fffe503bc9e3e03e50bf8e4fc90c9819d8 not found: ID does not exist"
Oct 30 01:36:27.609810 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.609753    6099 scope.go:122] "RemoveContainer" containerID="0cd549c7715b94409609f3a46322852f14bf19d0453a3c38429b380548840350"
Oct 30 01:36:27.610055 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:36:27.610025    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cd549c7715b94409609f3a46322852f14bf19d0453a3c38429b380548840350\": container with ID starting with 0cd549c7715b94409609f3a46322852f14bf19d0453a3c38429b380548840350 not found: ID does not exist" containerID="0cd549c7715b94409609f3a46322852f14bf19d0453a3c38429b380548840350"
Oct 30 01:36:27.610105 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.610065    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd549c7715b94409609f3a46322852f14bf19d0453a3c38429b380548840350"} err="failed to get container status \"0cd549c7715b94409609f3a46322852f14bf19d0453a3c38429b380548840350\": rpc error: code = NotFound desc = could not find container \"0cd549c7715b94409609f3a46322852f14bf19d0453a3c38429b380548840350\": container with ID starting with 0cd549c7715b94409609f3a46322852f14bf19d0453a3c38429b380548840350 not found: ID does not exist"
Oct 30 01:36:27.649068 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.649039    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"]
Oct 30 01:36:27.660779 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:27.660751    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7e040d-openshift-default-85b9d9878f65zcq"]
Oct 30 01:36:28.260653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:28.260624    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55fd1643-3498-4370-a0f8-a8af2204111f" path="/var/lib/kubelet/pods/55fd1643-3498-4370-a0f8-a8af2204111f/volumes"
Oct 30 01:36:28.871993 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:28.871909    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:36:29.173210 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:29.173128    6099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927" podUID="fff1cb9a-860c-4b1c-b59a-3246c8184e8d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:36:38.680224 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:38.680191    6099 generic.go:356] "Generic (PLEG): container finished" podID="fff1cb9a-860c-4b1c-b59a-3246c8184e8d" containerID="7e2787205ca816db8d8ed02333723bbf8b2170c19c800c416c998b215bfa3da2" exitCode=0
Oct 30 01:36:38.680621 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:38.680262    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927" event={"ID":"fff1cb9a-860c-4b1c-b59a-3246c8184e8d","Type":"ContainerDied","Data":"7e2787205ca816db8d8ed02333723bbf8b2170c19c800c416c998b215bfa3da2"}
Oct 30 01:36:38.763622 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:38.763598    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927"
Oct 30 01:36:38.803867 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:38.803839    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff1cb9a-860c-4b1c-b59a-3246c8184e8d-kube-api-access-rs2vm\" (UniqueName: \"kubernetes.io/projected/fff1cb9a-860c-4b1c-b59a-3246c8184e8d-kube-api-access-rs2vm\") pod \"fff1cb9a-860c-4b1c-b59a-3246c8184e8d\" (UID: \"fff1cb9a-860c-4b1c-b59a-3246c8184e8d\") "
Oct 30 01:36:38.804092 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:38.803978    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff1cb9a-860c-4b1c-b59a-3246c8184e8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fff1cb9a-860c-4b1c-b59a-3246c8184e8d-plugins-config-volume\") pod \"fff1cb9a-860c-4b1c-b59a-3246c8184e8d\" (UID: \"fff1cb9a-860c-4b1c-b59a-3246c8184e8d\") "
Oct 30 01:36:38.804817 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:38.804794    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff1cb9a-860c-4b1c-b59a-3246c8184e8d-plugins-config-volume" pod "fff1cb9a-860c-4b1c-b59a-3246c8184e8d" (UID: "fff1cb9a-860c-4b1c-b59a-3246c8184e8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:36:38.806485 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:38.806449    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff1cb9a-860c-4b1c-b59a-3246c8184e8d-kube-api-access-rs2vm" pod "fff1cb9a-860c-4b1c-b59a-3246c8184e8d" (UID: "fff1cb9a-860c-4b1c-b59a-3246c8184e8d"). InnerVolumeSpecName "kube-api-access-rs2vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:38.905621 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:38.905590    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs2vm\" (UniqueName: \"kubernetes.io/projected/fff1cb9a-860c-4b1c-b59a-3246c8184e8d-kube-api-access-rs2vm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:36:38.905621 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:38.905615    6099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fff1cb9a-860c-4b1c-b59a-3246c8184e8d-plugins-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:36:39.685927 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:39.685890    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927" event={"ID":"fff1cb9a-860c-4b1c-b59a-3246c8184e8d","Type":"ContainerDied","Data":"81a122ededccde379da9ca3684f7cdb73d92c519a753de6a763327b2bff0f255"}
Oct 30 01:36:39.686439 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:39.685956    6099 scope.go:122] "RemoveContainer" containerID="7e2787205ca816db8d8ed02333723bbf8b2170c19c800c416c998b215bfa3da2"
Oct 30 01:36:39.686439 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:39.685900    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927"
Oct 30 01:36:39.744352 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:39.744323    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927"]
Oct 30 01:36:39.756001 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:39.755973    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7e040d-6d656b46dsp927"]
Oct 30 01:36:40.260240 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:40.260201    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff1cb9a-860c-4b1c-b59a-3246c8184e8d" path="/var/lib/kubelet/pods/fff1cb9a-860c-4b1c-b59a-3246c8184e8d/volumes"
Oct 30 01:36:43.093657 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.093624    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"]
Oct 30 01:36:43.094280 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.094258    6099 cpu_manager.go:414] "containerMap: removing container" podUID="ba150208-a008-4eed-b889-a5cf36854373" containerName="maas-api"
Oct 30 01:36:43.094375 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.094282    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba150208-a008-4eed-b889-a5cf36854373" containerName="maas-api"
Oct 30 01:36:43.094375 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.094304    6099 cpu_manager.go:414] "containerMap: removing container" podUID="55fd1643-3498-4370-a0f8-a8af2204111f" containerName="istio-proxy"
Oct 30 01:36:43.094375 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.094313    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55fd1643-3498-4370-a0f8-a8af2204111f" containerName="istio-proxy"
Oct 30 01:36:43.094375 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.094343    6099 cpu_manager.go:414] "containerMap: removing container" podUID="fff1cb9a-860c-4b1c-b59a-3246c8184e8d" containerName="payload-processing"
Oct 30 01:36:43.094375 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.094350    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff1cb9a-860c-4b1c-b59a-3246c8184e8d" containerName="payload-processing"
Oct 30 01:36:43.094375 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.094365    6099 cpu_manager.go:414] "containerMap: removing container" podUID="55fd1643-3498-4370-a0f8-a8af2204111f" containerName="istio-proxy"
Oct 30 01:36:43.094375 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.094372    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55fd1643-3498-4370-a0f8-a8af2204111f" containerName="istio-proxy"
Oct 30 01:36:43.094775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.094488    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff1cb9a-860c-4b1c-b59a-3246c8184e8d" containerName="payload-processing"
Oct 30 01:36:43.094775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.094504    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="55fd1643-3498-4370-a0f8-a8af2204111f" containerName="istio-proxy"
Oct 30 01:36:43.094775 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.094523    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba150208-a008-4eed-b889-a5cf36854373" containerName="maas-api"
Oct 30 01:36:43.097784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.097763    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.125651 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.125624    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-5feb53af-kserve-self-signed-certs\""
Oct 30 01:36:43.127139 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.127117    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"]
Oct 30 01:36:43.148467 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.148437    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-dshm\") pod \"e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") " pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.148579 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.148477    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8572a245-3562-41a8-986b-a2866a38aaf8-tls-certs\") pod \"e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") " pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.148620 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.148577    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-home\") pod \"e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") " pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.148750 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.148728    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-tmp-dir\") pod \"e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") " pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.148813 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.148797    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-model-cache\") pod \"e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") " pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.148891 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.148873    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6hff\" (UniqueName: \"kubernetes.io/projected/8572a245-3562-41a8-986b-a2866a38aaf8-kube-api-access-d6hff\") pod \"e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") " pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.249885 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.249843    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6hff\" (UniqueName: \"kubernetes.io/projected/8572a245-3562-41a8-986b-a2866a38aaf8-kube-api-access-d6hff\") pod \"e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") " pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.249885 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.249889    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-dshm\") pod \"e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") " pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.250132 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.249916    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8572a245-3562-41a8-986b-a2866a38aaf8-tls-certs\") pod \"e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") " pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.250132 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.250019    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-home\") pod \"e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") " pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.250132 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.250100    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-tmp-dir\") pod \"e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") " pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.250132 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.250128    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-model-cache\") pod \"e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") " pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.250450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.250418    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-home\") pod \"e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") " pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.250566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.250462    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-model-cache\") pod \"e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") " pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.250566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.250500    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-tmp-dir\") pod \"e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") " pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.252140 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.252117    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-dshm\") pod \"e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") " pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.252782 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.252765    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8572a245-3562-41a8-986b-a2866a38aaf8-tls-certs\") pod \"e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") " pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.273062 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.273041    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6hff\" (UniqueName: \"kubernetes.io/projected/8572a245-3562-41a8-986b-a2866a38aaf8-kube-api-access-d6hff\") pod \"e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") " pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.410948 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.410869    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:43.706148 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:43.705797    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"]
Oct 30 01:36:43.708527 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:36:43.708498    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8572a245_3562_41a8_986b_a2866a38aaf8.slice/crio-d099d7a98ff11c5404a5cf6629afda54ef13abe6c675302ed291ebd62f6a8aa4 WatchSource:0}: Error finding container d099d7a98ff11c5404a5cf6629afda54ef13abe6c675302ed291ebd62f6a8aa4: Status 404 returned error can't find the container with id d099d7a98ff11c5404a5cf6629afda54ef13abe6c675302ed291ebd62f6a8aa4
Oct 30 01:36:44.716170 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:44.716137    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh" event={"ID":"8572a245-3562-41a8-986b-a2866a38aaf8","Type":"ContainerStarted","Data":"9e24a8b3bfb505d752263efc50642fb3c986bd9ff13101de93b0e009a69cf53c"}
Oct 30 01:36:44.716559 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:44.716182    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh" event={"ID":"8572a245-3562-41a8-986b-a2866a38aaf8","Type":"ContainerStarted","Data":"d099d7a98ff11c5404a5cf6629afda54ef13abe6c675302ed291ebd62f6a8aa4"}
Oct 30 01:36:44.767503 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:44.767444    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh" podStartSLOduration=1.767426791 podStartE2EDuration="1.767426791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:44.763990503 +0000 UTC m=+3698.620898259" watchObservedRunningTime="2026-10-30 01:36:44.767426791 +0000 UTC m=+3698.624334547"
Oct 30 01:36:45.411512 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:45.411475    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:45.555198 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:45.555147    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:36:46.633058 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:36:46.632987    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:36:53.411644 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:53.411614    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:53.424450 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:53.424422    6099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:36:53.784266 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:36:53.784184    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:37:00.375058 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:00.375025    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:37:01.411918 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:01.411879    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb"]
Oct 30 01:37:01.412702 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:01.412681    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="55fd1643-3498-4370-a0f8-a8af2204111f" containerName="istio-proxy"
Oct 30 01:37:01.416546 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:01.416525    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb"
Oct 30 01:37:01.433146 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:01.433120    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:37:01.454751 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:01.445037    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:37:01.454751 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:01.445109    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:37:01.454751 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:01.445695    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:37:01.454751 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:01.447652    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb"]
Oct 30 01:37:01.576907 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:01.576865    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stt4m\" (UniqueName: \"kubernetes.io/projected/5b7f095c-42e2-41a2-8789-a8838ecffe10-kube-api-access-stt4m\") pod \"maas-api-revoke-sub-452b6d7b-pppmb\" (UID: \"5b7f095c-42e2-41a2-8789-a8838ecffe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb"
Oct 30 01:37:01.577136 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:01.576922    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b7f095c-42e2-41a2-8789-a8838ecffe10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-pppmb\" (UID: \"5b7f095c-42e2-41a2-8789-a8838ecffe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb"
Oct 30 01:37:01.577136 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:01.576946    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b7f095c-42e2-41a2-8789-a8838ecffe10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-pppmb\" (UID: \"5b7f095c-42e2-41a2-8789-a8838ecffe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb"
Oct 30 01:37:01.678146 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:01.678043    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stt4m\" (UniqueName: \"kubernetes.io/projected/5b7f095c-42e2-41a2-8789-a8838ecffe10-kube-api-access-stt4m\") pod \"maas-api-revoke-sub-452b6d7b-pppmb\" (UID: \"5b7f095c-42e2-41a2-8789-a8838ecffe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb"
Oct 30 01:37:01.678146 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:01.678132    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b7f095c-42e2-41a2-8789-a8838ecffe10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-pppmb\" (UID: \"5b7f095c-42e2-41a2-8789-a8838ecffe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb"
Oct 30 01:37:01.678384 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:01.678358    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b7f095c-42e2-41a2-8789-a8838ecffe10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-pppmb\" (UID: \"5b7f095c-42e2-41a2-8789-a8838ecffe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb"
Oct 30 01:37:01.679239 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:01.679204    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b7f095c-42e2-41a2-8789-a8838ecffe10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-pppmb\" (UID: \"5b7f095c-42e2-41a2-8789-a8838ecffe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb"
Oct 30 01:37:01.680664 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:01.680641    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b7f095c-42e2-41a2-8789-a8838ecffe10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-pppmb\" (UID: \"5b7f095c-42e2-41a2-8789-a8838ecffe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb"
Oct 30 01:37:01.700917 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:01.700890    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stt4m\" (UniqueName: \"kubernetes.io/projected/5b7f095c-42e2-41a2-8789-a8838ecffe10-kube-api-access-stt4m\") pod \"maas-api-revoke-sub-452b6d7b-pppmb\" (UID: \"5b7f095c-42e2-41a2-8789-a8838ecffe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb"
Oct 30 01:37:01.735954 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:01.735927    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb"
Oct 30 01:37:02.045611 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:02.045572    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb"]
Oct 30 01:37:02.846306 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:02.846244    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb" event={"ID":"5b7f095c-42e2-41a2-8789-a8838ecffe10","Type":"ContainerStarted","Data":"1ce8ab800cb5106eefa90360acf18257451e4ff4e16cf7f49916ec971f132db4"}
Oct 30 01:37:02.846854 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:02.846338    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb" event={"ID":"5b7f095c-42e2-41a2-8789-a8838ecffe10","Type":"ContainerStarted","Data":"43bfae93b090c063bf1fecbb02fbdd380c0498903ea0301dbf763adbfca1fb2a"}
Oct 30 01:37:02.902729 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:02.895343    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb" podStartSLOduration=1.895324102 podStartE2EDuration="1.895324102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:02.893123577 +0000 UTC m=+3716.750031358" watchObservedRunningTime="2026-10-30 01:37:02.895324102 +0000 UTC m=+3716.752231860"
Oct 30 01:37:03.854857 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:03.854818    6099 generic.go:356] "Generic (PLEG): container finished" podID="5b7f095c-42e2-41a2-8789-a8838ecffe10" containerID="1ce8ab800cb5106eefa90360acf18257451e4ff4e16cf7f49916ec971f132db4" exitCode=0
Oct 30 01:37:03.855358 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:03.855018    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb" event={"ID":"5b7f095c-42e2-41a2-8789-a8838ecffe10","Type":"ContainerDied","Data":"1ce8ab800cb5106eefa90360acf18257451e4ff4e16cf7f49916ec971f132db4"}
Oct 30 01:37:04.997658 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:04.997634    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb"
Oct 30 01:37:05.055350 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:05.055321    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b7f095c-42e2-41a2-8789-a8838ecffe10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b7f095c-42e2-41a2-8789-a8838ecffe10-maas-api-xks-ca\") pod \"5b7f095c-42e2-41a2-8789-a8838ecffe10\" (UID: \"5b7f095c-42e2-41a2-8789-a8838ecffe10\") "
Oct 30 01:37:05.055487 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:05.055372    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7f095c-42e2-41a2-8789-a8838ecffe10-kube-api-access-stt4m\" (UniqueName: \"kubernetes.io/projected/5b7f095c-42e2-41a2-8789-a8838ecffe10-kube-api-access-stt4m\") pod \"5b7f095c-42e2-41a2-8789-a8838ecffe10\" (UID: \"5b7f095c-42e2-41a2-8789-a8838ecffe10\") "
Oct 30 01:37:05.055487 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:05.055421    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b7f095c-42e2-41a2-8789-a8838ecffe10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b7f095c-42e2-41a2-8789-a8838ecffe10-maas-api-openshift-ca\") pod \"5b7f095c-42e2-41a2-8789-a8838ecffe10\" (UID: \"5b7f095c-42e2-41a2-8789-a8838ecffe10\") "
Oct 30 01:37:05.055844 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:05.055818    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7f095c-42e2-41a2-8789-a8838ecffe10-maas-api-openshift-ca" pod "5b7f095c-42e2-41a2-8789-a8838ecffe10" (UID: "5b7f095c-42e2-41a2-8789-a8838ecffe10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:37:05.057568 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:05.057546    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7f095c-42e2-41a2-8789-a8838ecffe10-kube-api-access-stt4m" pod "5b7f095c-42e2-41a2-8789-a8838ecffe10" (UID: "5b7f095c-42e2-41a2-8789-a8838ecffe10"). InnerVolumeSpecName "kube-api-access-stt4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:37:05.057568 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:05.057550    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b7f095c-42e2-41a2-8789-a8838ecffe10-maas-api-xks-ca" pod "5b7f095c-42e2-41a2-8789-a8838ecffe10" (UID: "5b7f095c-42e2-41a2-8789-a8838ecffe10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:37:05.156959 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:05.156880    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b7f095c-42e2-41a2-8789-a8838ecffe10-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:37:05.156959 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:05.156911    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stt4m\" (UniqueName: \"kubernetes.io/projected/5b7f095c-42e2-41a2-8789-a8838ecffe10-kube-api-access-stt4m\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:37:05.156959 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:05.156921    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b7f095c-42e2-41a2-8789-a8838ecffe10-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:37:05.867360 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:05.867328    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb" event={"ID":"5b7f095c-42e2-41a2-8789-a8838ecffe10","Type":"ContainerDied","Data":"43bfae93b090c063bf1fecbb02fbdd380c0498903ea0301dbf763adbfca1fb2a"}
Oct 30 01:37:05.867360 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:05.867347    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-pppmb"
Oct 30 01:37:05.867594 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:05.867381    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43bfae93b090c063bf1fecbb02fbdd380c0498903ea0301dbf763adbfca1fb2a"
Oct 30 01:37:14.561225 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:14.561192    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:37:15.958815 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:15.958768    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"]
Oct 30 01:37:16.014292 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:16.014252    6099 scope.go:122] "RemoveContainer" containerID="d76be53431b1c7fbe68dac7e545e42c4a380b3faede222087db97f74fa28ffbd"
Oct 30 01:37:16.025988 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:16.025961    6099 scope.go:122] "RemoveContainer" containerID="d4d9af4563606a0d16b1cfddb46cdcfdba10c960a10ab2f50dc9d7923e6e5c2b"
Oct 30 01:37:16.036282 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:16.036252    6099 scope.go:122] "RemoveContainer" containerID="c49013c332719ef5143ff60c8a340ed923baeefa454ccadf1db3082f2b61e785"
Oct 30 01:37:16.661277 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:37:16.661238    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:37:19.618042 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.618001    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg"]
Oct 30 01:37:19.618784 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.618693    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5b7f095c-42e2-41a2-8789-a8838ecffe10" containerName="revoke-keys"
Oct 30 01:37:19.618863 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.618791    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b7f095c-42e2-41a2-8789-a8838ecffe10" containerName="revoke-keys"
Oct 30 01:37:19.618993 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.618978    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b7f095c-42e2-41a2-8789-a8838ecffe10" containerName="revoke-keys"
Oct 30 01:37:19.624229 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.624207    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg"
Oct 30 01:37:19.653386 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.653353    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg"]
Oct 30 01:37:19.653961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.653936    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:37:19.654111 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.653960    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:37:19.654111 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.654022    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:37:19.662810 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.662779    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:37:19.678296 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.678266    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbqjd\" (UniqueName: \"kubernetes.io/projected/c377dff7-9004-41c2-b1bf-1107b2ab617c-kube-api-access-sbqjd\") pod \"maas-api-revoke-sub-3e31a506-plsxg\" (UID: \"c377dff7-9004-41c2-b1bf-1107b2ab617c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg"
Oct 30 01:37:19.678504 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.678485    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c377dff7-9004-41c2-b1bf-1107b2ab617c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-plsxg\" (UID: \"c377dff7-9004-41c2-b1bf-1107b2ab617c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg"
Oct 30 01:37:19.678780 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.678758    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c377dff7-9004-41c2-b1bf-1107b2ab617c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-plsxg\" (UID: \"c377dff7-9004-41c2-b1bf-1107b2ab617c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg"
Oct 30 01:37:19.779913 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.779873    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbqjd\" (UniqueName: \"kubernetes.io/projected/c377dff7-9004-41c2-b1bf-1107b2ab617c-kube-api-access-sbqjd\") pod \"maas-api-revoke-sub-3e31a506-plsxg\" (UID: \"c377dff7-9004-41c2-b1bf-1107b2ab617c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg"
Oct 30 01:37:19.779913 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.779919    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c377dff7-9004-41c2-b1bf-1107b2ab617c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-plsxg\" (UID: \"c377dff7-9004-41c2-b1bf-1107b2ab617c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg"
Oct 30 01:37:19.780193 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.780041    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c377dff7-9004-41c2-b1bf-1107b2ab617c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-plsxg\" (UID: \"c377dff7-9004-41c2-b1bf-1107b2ab617c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg"
Oct 30 01:37:19.780816 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.780789    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c377dff7-9004-41c2-b1bf-1107b2ab617c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-plsxg\" (UID: \"c377dff7-9004-41c2-b1bf-1107b2ab617c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg"
Oct 30 01:37:19.782526 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.782497    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c377dff7-9004-41c2-b1bf-1107b2ab617c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-plsxg\" (UID: \"c377dff7-9004-41c2-b1bf-1107b2ab617c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg"
Oct 30 01:37:19.803076 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.803043    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbqjd\" (UniqueName: \"kubernetes.io/projected/c377dff7-9004-41c2-b1bf-1107b2ab617c-kube-api-access-sbqjd\") pod \"maas-api-revoke-sub-3e31a506-plsxg\" (UID: \"c377dff7-9004-41c2-b1bf-1107b2ab617c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg"
Oct 30 01:37:19.938388 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:19.938294    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg"
Oct 30 01:37:20.251336 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:20.251294    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg"]
Oct 30 01:37:20.252955 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:37:20.252925    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc377dff7_9004_41c2_b1bf_1107b2ab617c.slice/crio-2c15091cb393a3e02e9b75e1b7177987dd3c74039329708915e48cd2bd91d42b WatchSource:0}: Error finding container 2c15091cb393a3e02e9b75e1b7177987dd3c74039329708915e48cd2bd91d42b: Status 404 returned error can't find the container with id 2c15091cb393a3e02e9b75e1b7177987dd3c74039329708915e48cd2bd91d42b
Oct 30 01:37:20.972098 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:20.972053    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg" event={"ID":"c377dff7-9004-41c2-b1bf-1107b2ab617c","Type":"ContainerStarted","Data":"ae9274f6ed57c766ae6bf6817e9dfbd185dbacdca30bbbee1e2ee7fad3c8453e"}
Oct 30 01:37:20.972691 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:20.972666    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg" event={"ID":"c377dff7-9004-41c2-b1bf-1107b2ab617c","Type":"ContainerStarted","Data":"2c15091cb393a3e02e9b75e1b7177987dd3c74039329708915e48cd2bd91d42b"}
Oct 30 01:37:21.037914 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:21.037840    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg" podStartSLOduration=2.037821687 podStartE2EDuration="2.037821687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:21.024287633 +0000 UTC m=+3734.881195413" watchObservedRunningTime="2026-10-30 01:37:21.037821687 +0000 UTC m=+3734.894729443"
Oct 30 01:37:21.993171 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:21.993135    6099 generic.go:356] "Generic (PLEG): container finished" podID="c377dff7-9004-41c2-b1bf-1107b2ab617c" containerID="ae9274f6ed57c766ae6bf6817e9dfbd185dbacdca30bbbee1e2ee7fad3c8453e" exitCode=0
Oct 30 01:37:22.003727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:21.993787    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg" event={"ID":"c377dff7-9004-41c2-b1bf-1107b2ab617c","Type":"ContainerDied","Data":"ae9274f6ed57c766ae6bf6817e9dfbd185dbacdca30bbbee1e2ee7fad3c8453e"}
Oct 30 01:37:22.824029 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:22.823995    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:37:23.138192 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:23.138162    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg"
Oct 30 01:37:23.152915 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:23.152895    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c377dff7-9004-41c2-b1bf-1107b2ab617c-kube-api-access-sbqjd\" (UniqueName: \"kubernetes.io/projected/c377dff7-9004-41c2-b1bf-1107b2ab617c-kube-api-access-sbqjd\") pod \"c377dff7-9004-41c2-b1bf-1107b2ab617c\" (UID: \"c377dff7-9004-41c2-b1bf-1107b2ab617c\") "
Oct 30 01:37:23.152997 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:23.152940    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c377dff7-9004-41c2-b1bf-1107b2ab617c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c377dff7-9004-41c2-b1bf-1107b2ab617c-maas-api-xks-ca\") pod \"c377dff7-9004-41c2-b1bf-1107b2ab617c\" (UID: \"c377dff7-9004-41c2-b1bf-1107b2ab617c\") "
Oct 30 01:37:23.153047 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:23.153020    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c377dff7-9004-41c2-b1bf-1107b2ab617c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c377dff7-9004-41c2-b1bf-1107b2ab617c-maas-api-openshift-ca\") pod \"c377dff7-9004-41c2-b1bf-1107b2ab617c\" (UID: \"c377dff7-9004-41c2-b1bf-1107b2ab617c\") "
Oct 30 01:37:23.153416 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:23.153391    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c377dff7-9004-41c2-b1bf-1107b2ab617c-maas-api-openshift-ca" pod "c377dff7-9004-41c2-b1bf-1107b2ab617c" (UID: "c377dff7-9004-41c2-b1bf-1107b2ab617c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:37:23.155040 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:23.155014    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c377dff7-9004-41c2-b1bf-1107b2ab617c-maas-api-xks-ca" pod "c377dff7-9004-41c2-b1bf-1107b2ab617c" (UID: "c377dff7-9004-41c2-b1bf-1107b2ab617c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:37:23.155120 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:23.155073    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c377dff7-9004-41c2-b1bf-1107b2ab617c-kube-api-access-sbqjd" pod "c377dff7-9004-41c2-b1bf-1107b2ab617c" (UID: "c377dff7-9004-41c2-b1bf-1107b2ab617c"). InnerVolumeSpecName "kube-api-access-sbqjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:37:23.254019 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:23.253984    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbqjd\" (UniqueName: \"kubernetes.io/projected/c377dff7-9004-41c2-b1bf-1107b2ab617c-kube-api-access-sbqjd\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:37:23.254019 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:23.254016    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c377dff7-9004-41c2-b1bf-1107b2ab617c-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:37:23.254019 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:23.254025    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c377dff7-9004-41c2-b1bf-1107b2ab617c-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:37:24.005401 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:24.005369    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg"
Oct 30 01:37:24.005401 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:24.005379    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-plsxg" event={"ID":"c377dff7-9004-41c2-b1bf-1107b2ab617c","Type":"ContainerDied","Data":"2c15091cb393a3e02e9b75e1b7177987dd3c74039329708915e48cd2bd91d42b"}
Oct 30 01:37:24.005668 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:24.005423    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c15091cb393a3e02e9b75e1b7177987dd3c74039329708915e48cd2bd91d42b"
Oct 30 01:37:30.853144 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:30.853108    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:37:30.982613 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:30.982557    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh" podUID="8572a245-3562-41a8-986b-a2866a38aaf8" containerName="main" containerID="cri-o://9e24a8b3bfb505d752263efc50642fb3c986bd9ff13101de93b0e009a69cf53c" gracePeriod=45
Oct 30 01:37:31.252166 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.252127    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:37:31.339735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.335028    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-dshm\") pod \"8572a245-3562-41a8-986b-a2866a38aaf8\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") "
Oct 30 01:37:31.339735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.335129    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8572a245-3562-41a8-986b-a2866a38aaf8-kube-api-access-d6hff\" (UniqueName: \"kubernetes.io/projected/8572a245-3562-41a8-986b-a2866a38aaf8-kube-api-access-d6hff\") pod \"8572a245-3562-41a8-986b-a2866a38aaf8\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") "
Oct 30 01:37:31.339735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.335174    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8572a245-3562-41a8-986b-a2866a38aaf8-tls-certs\" (UniqueName: \"kubernetes.io/secret/8572a245-3562-41a8-986b-a2866a38aaf8-tls-certs\") pod \"8572a245-3562-41a8-986b-a2866a38aaf8\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") "
Oct 30 01:37:31.339735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.335268    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-home\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-home\") pod \"8572a245-3562-41a8-986b-a2866a38aaf8\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") "
Oct 30 01:37:31.339735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.335342    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-model-cache\") pod \"8572a245-3562-41a8-986b-a2866a38aaf8\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") "
Oct 30 01:37:31.339735 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.335379    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-tmp-dir\") pod \"8572a245-3562-41a8-986b-a2866a38aaf8\" (UID: \"8572a245-3562-41a8-986b-a2866a38aaf8\") "
Oct 30 01:37:31.363027 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.362210    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-tmp-dir" pod "8572a245-3562-41a8-986b-a2866a38aaf8" (UID: "8572a245-3562-41a8-986b-a2866a38aaf8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:37:31.363470 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.363263    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-model-cache" pod "8572a245-3562-41a8-986b-a2866a38aaf8" (UID: "8572a245-3562-41a8-986b-a2866a38aaf8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:37:31.363727 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.363636    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-home" pod "8572a245-3562-41a8-986b-a2866a38aaf8" (UID: "8572a245-3562-41a8-986b-a2866a38aaf8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:37:31.416050 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.416002    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8572a245-3562-41a8-986b-a2866a38aaf8-tls-certs" pod "8572a245-3562-41a8-986b-a2866a38aaf8" (UID: "8572a245-3562-41a8-986b-a2866a38aaf8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:37:31.440889 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.440855    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8572a245-3562-41a8-986b-a2866a38aaf8-kube-api-access-d6hff" pod "8572a245-3562-41a8-986b-a2866a38aaf8" (UID: "8572a245-3562-41a8-986b-a2866a38aaf8"). InnerVolumeSpecName "kube-api-access-d6hff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:37:31.441278 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.441254    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-dshm" pod "8572a245-3562-41a8-986b-a2866a38aaf8" (UID: "8572a245-3562-41a8-986b-a2866a38aaf8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:37:31.460777 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.460751    6099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:37:31.460777 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.460774    6099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:37:31.460926 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.460786    6099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:37:31.460926 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.460793    6099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8572a245-3562-41a8-986b-a2866a38aaf8-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:37:31.460926 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.460801    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6hff\" (UniqueName: \"kubernetes.io/projected/8572a245-3562-41a8-986b-a2866a38aaf8-kube-api-access-d6hff\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:37:31.460926 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:31.460812    6099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8572a245-3562-41a8-986b-a2866a38aaf8-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:37:32.048018 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:32.047982    6099 generic.go:356] "Generic (PLEG): container finished" podID="8572a245-3562-41a8-986b-a2866a38aaf8" containerID="9e24a8b3bfb505d752263efc50642fb3c986bd9ff13101de93b0e009a69cf53c" exitCode=0
Oct 30 01:37:32.048018 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:32.048018    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"
Oct 30 01:37:32.048614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:32.048066    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh" event={"ID":"8572a245-3562-41a8-986b-a2866a38aaf8","Type":"ContainerDied","Data":"9e24a8b3bfb505d752263efc50642fb3c986bd9ff13101de93b0e009a69cf53c"}
Oct 30 01:37:32.048614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:32.048117    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh" event={"ID":"8572a245-3562-41a8-986b-a2866a38aaf8","Type":"ContainerDied","Data":"d099d7a98ff11c5404a5cf6629afda54ef13abe6c675302ed291ebd62f6a8aa4"}
Oct 30 01:37:32.048614 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:32.048142    6099 scope.go:122] "RemoveContainer" containerID="9e24a8b3bfb505d752263efc50642fb3c986bd9ff13101de93b0e009a69cf53c"
Oct 30 01:37:32.057770 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:32.057754    6099 scope.go:122] "RemoveContainer" containerID="9e24a8b3bfb505d752263efc50642fb3c986bd9ff13101de93b0e009a69cf53c"
Oct 30 01:37:32.058005 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:37:32.057985    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e24a8b3bfb505d752263efc50642fb3c986bd9ff13101de93b0e009a69cf53c\": container with ID starting with 9e24a8b3bfb505d752263efc50642fb3c986bd9ff13101de93b0e009a69cf53c not found: ID does not exist" containerID="9e24a8b3bfb505d752263efc50642fb3c986bd9ff13101de93b0e009a69cf53c"
Oct 30 01:37:32.058061 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:32.058016    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e24a8b3bfb505d752263efc50642fb3c986bd9ff13101de93b0e009a69cf53c"} err="failed to get container status \"9e24a8b3bfb505d752263efc50642fb3c986bd9ff13101de93b0e009a69cf53c\": rpc error: code = NotFound desc = could not find container \"9e24a8b3bfb505d752263efc50642fb3c986bd9ff13101de93b0e009a69cf53c\": container with ID starting with 9e24a8b3bfb505d752263efc50642fb3c986bd9ff13101de93b0e009a69cf53c not found: ID does not exist"
Oct 30 01:37:32.113382 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:32.113350    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"]
Oct 30 01:37:32.126093 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:32.126067    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-5feb53af-kserve-79959b7cd9-kk4bh"]
Oct 30 01:37:32.260808 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:32.260776    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8572a245-3562-41a8-986b-a2866a38aaf8" path="/var/lib/kubelet/pods/8572a245-3562-41a8-986b-a2866a38aaf8/volumes"
Oct 30 01:37:35.991993 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:35.991958    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:37:42.091096 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.091057    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh"]
Oct 30 01:37:42.091750 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.091696    6099 cpu_manager.go:414] "containerMap: removing container" podUID="8572a245-3562-41a8-986b-a2866a38aaf8" containerName="main"
Oct 30 01:37:42.091750 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.091749    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8572a245-3562-41a8-986b-a2866a38aaf8" containerName="main"
Oct 30 01:37:42.091935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.091769    6099 cpu_manager.go:414] "containerMap: removing container" podUID="c377dff7-9004-41c2-b1bf-1107b2ab617c" containerName="revoke-keys"
Oct 30 01:37:42.091935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.091776    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c377dff7-9004-41c2-b1bf-1107b2ab617c" containerName="revoke-keys"
Oct 30 01:37:42.091935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.091898    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="8572a245-3562-41a8-986b-a2866a38aaf8" containerName="main"
Oct 30 01:37:42.091935 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.091912    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c377dff7-9004-41c2-b1bf-1107b2ab617c" containerName="revoke-keys"
Oct 30 01:37:42.096788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.096766    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh"
Oct 30 01:37:42.124179 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.124148    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:37:42.124179 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.124151    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:37:42.124398 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.124248    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pkbqp\""
Oct 30 01:37:42.124398 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.124359    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:37:42.125986 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.125960    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh"]
Oct 30 01:37:42.238801 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.238765    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcdq6\" (UniqueName: \"kubernetes.io/projected/d1362a15-d730-483f-bd3d-b83738bdf704-kube-api-access-zcdq6\") pod \"maas-api-revoke-sub-35ff1eed-swmnh\" (UID: \"d1362a15-d730-483f-bd3d-b83738bdf704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh"
Oct 30 01:37:42.238980 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.238890    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1362a15-d730-483f-bd3d-b83738bdf704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35ff1eed-swmnh\" (UID: \"d1362a15-d730-483f-bd3d-b83738bdf704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh"
Oct 30 01:37:42.238980 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.238938    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1362a15-d730-483f-bd3d-b83738bdf704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35ff1eed-swmnh\" (UID: \"d1362a15-d730-483f-bd3d-b83738bdf704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh"
Oct 30 01:37:42.339905 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.339870    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1362a15-d730-483f-bd3d-b83738bdf704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35ff1eed-swmnh\" (UID: \"d1362a15-d730-483f-bd3d-b83738bdf704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh"
Oct 30 01:37:42.340054 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.339917    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1362a15-d730-483f-bd3d-b83738bdf704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35ff1eed-swmnh\" (UID: \"d1362a15-d730-483f-bd3d-b83738bdf704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh"
Oct 30 01:37:42.340054 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.339981    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcdq6\" (UniqueName: \"kubernetes.io/projected/d1362a15-d730-483f-bd3d-b83738bdf704-kube-api-access-zcdq6\") pod \"maas-api-revoke-sub-35ff1eed-swmnh\" (UID: \"d1362a15-d730-483f-bd3d-b83738bdf704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh"
Oct 30 01:37:42.340590 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.340562    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1362a15-d730-483f-bd3d-b83738bdf704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35ff1eed-swmnh\" (UID: \"d1362a15-d730-483f-bd3d-b83738bdf704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh"
Oct 30 01:37:42.342271 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.342221    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1362a15-d730-483f-bd3d-b83738bdf704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35ff1eed-swmnh\" (UID: \"d1362a15-d730-483f-bd3d-b83738bdf704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh"
Oct 30 01:37:42.362525 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.362495    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcdq6\" (UniqueName: \"kubernetes.io/projected/d1362a15-d730-483f-bd3d-b83738bdf704-kube-api-access-zcdq6\") pod \"maas-api-revoke-sub-35ff1eed-swmnh\" (UID: \"d1362a15-d730-483f-bd3d-b83738bdf704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh"
Oct 30 01:37:42.410594 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.410571    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh"
Oct 30 01:37:42.704188 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:42.704163    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh"]
Oct 30 01:37:42.706064 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:37:42.706035    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1362a15_d730_483f_bd3d_b83738bdf704.slice/crio-d98f68d89c49ecb68bcbffe47104a4881c28a2be2337581b4ade2b3de58a6c17 WatchSource:0}: Error finding container d98f68d89c49ecb68bcbffe47104a4881c28a2be2337581b4ade2b3de58a6c17: Status 404 returned error can't find the container with id d98f68d89c49ecb68bcbffe47104a4881c28a2be2337581b4ade2b3de58a6c17
Oct 30 01:37:43.127597 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:43.127556    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh" event={"ID":"d1362a15-d730-483f-bd3d-b83738bdf704","Type":"ContainerStarted","Data":"31fe4afbfdbfe0bd57f8826ab4303be8a25060e2c6beca3f3d1935a972c88bc9"}
Oct 30 01:37:43.128035 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:43.127619    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh" event={"ID":"d1362a15-d730-483f-bd3d-b83738bdf704","Type":"ContainerStarted","Data":"d98f68d89c49ecb68bcbffe47104a4881c28a2be2337581b4ade2b3de58a6c17"}
Oct 30 01:37:43.170560 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:43.170506    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh" podStartSLOduration=1.170491524 podStartE2EDuration="1.170491524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:43.169251156 +0000 UTC m=+3757.026158911" watchObservedRunningTime="2026-10-30 01:37:43.170491524 +0000 UTC m=+3757.027399300"
Oct 30 01:37:44.133865 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:44.133825    6099 generic.go:356] "Generic (PLEG): container finished" podID="d1362a15-d730-483f-bd3d-b83738bdf704" containerID="31fe4afbfdbfe0bd57f8826ab4303be8a25060e2c6beca3f3d1935a972c88bc9" exitCode=0
Oct 30 01:37:44.134271 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:44.133905    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh" event={"ID":"d1362a15-d730-483f-bd3d-b83738bdf704","Type":"ContainerDied","Data":"31fe4afbfdbfe0bd57f8826ab4303be8a25060e2c6beca3f3d1935a972c88bc9"}
Oct 30 01:37:45.262156 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:45.262133    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh"
Oct 30 01:37:45.370612 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:45.370578    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1362a15-d730-483f-bd3d-b83738bdf704-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1362a15-d730-483f-bd3d-b83738bdf704-maas-api-openshift-ca\") pod \"d1362a15-d730-483f-bd3d-b83738bdf704\" (UID: \"d1362a15-d730-483f-bd3d-b83738bdf704\") "
Oct 30 01:37:45.371134 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:45.370632    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1362a15-d730-483f-bd3d-b83738bdf704-kube-api-access-zcdq6\" (UniqueName: \"kubernetes.io/projected/d1362a15-d730-483f-bd3d-b83738bdf704-kube-api-access-zcdq6\") pod \"d1362a15-d730-483f-bd3d-b83738bdf704\" (UID: \"d1362a15-d730-483f-bd3d-b83738bdf704\") "
Oct 30 01:37:45.371134 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:45.370780    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1362a15-d730-483f-bd3d-b83738bdf704-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1362a15-d730-483f-bd3d-b83738bdf704-maas-api-xks-ca\") pod \"d1362a15-d730-483f-bd3d-b83738bdf704\" (UID: \"d1362a15-d730-483f-bd3d-b83738bdf704\") "
Oct 30 01:37:45.371134 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:45.371033    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1362a15-d730-483f-bd3d-b83738bdf704-maas-api-openshift-ca" pod "d1362a15-d730-483f-bd3d-b83738bdf704" (UID: "d1362a15-d730-483f-bd3d-b83738bdf704"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:37:45.371519 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:45.371495    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1362a15-d730-483f-bd3d-b83738bdf704-maas-api-openshift-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:37:45.373318 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:45.373295    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1362a15-d730-483f-bd3d-b83738bdf704-maas-api-xks-ca" pod "d1362a15-d730-483f-bd3d-b83738bdf704" (UID: "d1362a15-d730-483f-bd3d-b83738bdf704"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:37:45.373404 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:45.373315    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1362a15-d730-483f-bd3d-b83738bdf704-kube-api-access-zcdq6" pod "d1362a15-d730-483f-bd3d-b83738bdf704" (UID: "d1362a15-d730-483f-bd3d-b83738bdf704"). InnerVolumeSpecName "kube-api-access-zcdq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:37:45.472080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:45.472013    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1362a15-d730-483f-bd3d-b83738bdf704-maas-api-xks-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:37:45.472080 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:45.472040    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcdq6\" (UniqueName: \"kubernetes.io/projected/d1362a15-d730-483f-bd3d-b83738bdf704-kube-api-access-zcdq6\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:37:46.144155 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:46.144125    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh"
Oct 30 01:37:46.144359 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:46.144161    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ff1eed-swmnh" event={"ID":"d1362a15-d730-483f-bd3d-b83738bdf704","Type":"ContainerDied","Data":"d98f68d89c49ecb68bcbffe47104a4881c28a2be2337581b4ade2b3de58a6c17"}
Oct 30 01:37:46.144359 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:46.144211    6099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d98f68d89c49ecb68bcbffe47104a4881c28a2be2337581b4ade2b3de58a6c17"
Oct 30 01:37:46.628009 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:37:46.627973    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:37:53.329115 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:53.329083    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:37:54.194451 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.194415    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"]
Oct 30 01:37:54.194998 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.194975    6099 cpu_manager.go:414] "containerMap: removing container" podUID="d1362a15-d730-483f-bd3d-b83738bdf704" containerName="revoke-keys"
Oct 30 01:37:54.194998 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.194996    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1362a15-d730-483f-bd3d-b83738bdf704" containerName="revoke-keys"
Oct 30 01:37:54.195180 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.195075    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1362a15-d730-483f-bd3d-b83738bdf704" containerName="revoke-keys"
Oct 30 01:37:54.198279 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.198258    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"
Oct 30 01:37:54.214243 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.214218    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"]
Oct 30 01:37:54.217769 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.217750    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"
Oct 30 01:37:54.251155 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.251124    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:37:54.251337 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.251323    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k5bc5\""
Oct 30 01:37:54.251409 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.251387    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:37:54.251598 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.251574    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 01:37:54.251802 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.251606    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 01:37:54.253169 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.253143    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"]
Oct 30 01:37:54.253663 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.253639    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-maas-api-tls\") pod \"maas-api-5c48877c75-nkchq\" (UID: \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"
Oct 30 01:37:54.253769 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.253687    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxh2m\" (UniqueName: \"kubernetes.io/projected/9aa787cc-9150-483f-b4bf-968feda47917-kube-api-access-gxh2m\") pod \"maas-api-595cc648c-h6g9c\" (UID: \"9aa787cc-9150-483f-b4bf-968feda47917\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"
Oct 30 01:37:54.253912 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.253875    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-metrics-certs\") pod \"maas-api-5c48877c75-nkchq\" (UID: \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"
Oct 30 01:37:54.254003 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.253922    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-openshift-service-ca\") pod \"maas-api-5c48877c75-nkchq\" (UID: \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"
Oct 30 01:37:54.254003 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.253949    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp9hl\" (UniqueName: \"kubernetes.io/projected/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-kube-api-access-mp9hl\") pod \"maas-api-5c48877c75-nkchq\" (UID: \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"
Oct 30 01:37:54.254003 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.253976    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa787cc-9150-483f-b4bf-968feda47917-openshift-service-ca\") pod \"maas-api-595cc648c-h6g9c\" (UID: \"9aa787cc-9150-483f-b4bf-968feda47917\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"
Oct 30 01:37:54.254189 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.254007    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa787cc-9150-483f-b4bf-968feda47917-metrics-certs\") pod \"maas-api-595cc648c-h6g9c\" (UID: \"9aa787cc-9150-483f-b4bf-968feda47917\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"
Oct 30 01:37:54.254189 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.254089    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa787cc-9150-483f-b4bf-968feda47917-maas-api-tls\") pod \"maas-api-595cc648c-h6g9c\" (UID: \"9aa787cc-9150-483f-b4bf-968feda47917\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"
Oct 30 01:37:54.260283 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.260260    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"]
Oct 30 01:37:54.354564 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.354531    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa787cc-9150-483f-b4bf-968feda47917-maas-api-tls\") pod \"maas-api-595cc648c-h6g9c\" (UID: \"9aa787cc-9150-483f-b4bf-968feda47917\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"
Oct 30 01:37:54.354564 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.354566    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-maas-api-tls\") pod \"maas-api-5c48877c75-nkchq\" (UID: \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"
Oct 30 01:37:54.355121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.354597    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxh2m\" (UniqueName: \"kubernetes.io/projected/9aa787cc-9150-483f-b4bf-968feda47917-kube-api-access-gxh2m\") pod \"maas-api-595cc648c-h6g9c\" (UID: \"9aa787cc-9150-483f-b4bf-968feda47917\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"
Oct 30 01:37:54.355121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.354755    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-metrics-certs\") pod \"maas-api-5c48877c75-nkchq\" (UID: \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"
Oct 30 01:37:54.355121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.354791    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-openshift-service-ca\") pod \"maas-api-5c48877c75-nkchq\" (UID: \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"
Oct 30 01:37:54.355121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.354821    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp9hl\" (UniqueName: \"kubernetes.io/projected/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-kube-api-access-mp9hl\") pod \"maas-api-5c48877c75-nkchq\" (UID: \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"
Oct 30 01:37:54.355121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.354846    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa787cc-9150-483f-b4bf-968feda47917-openshift-service-ca\") pod \"maas-api-595cc648c-h6g9c\" (UID: \"9aa787cc-9150-483f-b4bf-968feda47917\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"
Oct 30 01:37:54.355121 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.354874    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa787cc-9150-483f-b4bf-968feda47917-metrics-certs\") pod \"maas-api-595cc648c-h6g9c\" (UID: \"9aa787cc-9150-483f-b4bf-968feda47917\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"
Oct 30 01:37:54.355679 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.355652    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-openshift-service-ca\") pod \"maas-api-5c48877c75-nkchq\" (UID: \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"
Oct 30 01:37:54.355817 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.355788    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa787cc-9150-483f-b4bf-968feda47917-openshift-service-ca\") pod \"maas-api-595cc648c-h6g9c\" (UID: \"9aa787cc-9150-483f-b4bf-968feda47917\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"
Oct 30 01:37:54.357216 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.357193    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa787cc-9150-483f-b4bf-968feda47917-maas-api-tls\") pod \"maas-api-595cc648c-h6g9c\" (UID: \"9aa787cc-9150-483f-b4bf-968feda47917\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"
Oct 30 01:37:54.357323 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.357300    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa787cc-9150-483f-b4bf-968feda47917-metrics-certs\") pod \"maas-api-595cc648c-h6g9c\" (UID: \"9aa787cc-9150-483f-b4bf-968feda47917\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"
Oct 30 01:37:54.357496 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.357480    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-maas-api-tls\") pod \"maas-api-5c48877c75-nkchq\" (UID: \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"
Oct 30 01:37:54.357556 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.357539    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-metrics-certs\") pod \"maas-api-5c48877c75-nkchq\" (UID: \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"
Oct 30 01:37:54.377480 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.377457    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxh2m\" (UniqueName: \"kubernetes.io/projected/9aa787cc-9150-483f-b4bf-968feda47917-kube-api-access-gxh2m\") pod \"maas-api-595cc648c-h6g9c\" (UID: \"9aa787cc-9150-483f-b4bf-968feda47917\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"
Oct 30 01:37:54.377576 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.377548    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp9hl\" (UniqueName: \"kubernetes.io/projected/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-kube-api-access-mp9hl\") pod \"maas-api-5c48877c75-nkchq\" (UID: \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"
Oct 30 01:37:54.512968 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.512894    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"
Oct 30 01:37:54.533249 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.533222    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"
Oct 30 01:37:54.813887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.813855    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"]
Oct 30 01:37:54.815487 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:37:54.815406    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9b5b01_f0d1_428c_b005_1fc0aabda0d3.slice/crio-a16f1eecf904104f40fc0448c8634700dde025810266cd3dc40122d6603ebb93 WatchSource:0}: Error finding container a16f1eecf904104f40fc0448c8634700dde025810266cd3dc40122d6603ebb93: Status 404 returned error can't find the container with id a16f1eecf904104f40fc0448c8634700dde025810266cd3dc40122d6603ebb93
Oct 30 01:37:54.816833 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.816814    6099 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:37:54.848639 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:54.848616    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"]
Oct 30 01:37:54.849888 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:37:54.849862    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa787cc_9150_483f_b4bf_968feda47917.slice/crio-7158677347e90b3b76b09441d74b30dead7c9f5313cb7bed09a884c05143ddff WatchSource:0}: Error finding container 7158677347e90b3b76b09441d74b30dead7c9f5313cb7bed09a884c05143ddff: Status 404 returned error can't find the container with id 7158677347e90b3b76b09441d74b30dead7c9f5313cb7bed09a884c05143ddff
Oct 30 01:37:55.189755 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:55.189660    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c" event={"ID":"9aa787cc-9150-483f-b4bf-968feda47917","Type":"ContainerStarted","Data":"7158677347e90b3b76b09441d74b30dead7c9f5313cb7bed09a884c05143ddff"}
Oct 30 01:37:55.190860 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:55.190835    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq" event={"ID":"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3","Type":"ContainerStarted","Data":"a16f1eecf904104f40fc0448c8634700dde025810266cd3dc40122d6603ebb93"}
Oct 30 01:37:56.197562 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:56.197527    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c" event={"ID":"9aa787cc-9150-483f-b4bf-968feda47917","Type":"ContainerStarted","Data":"c42e79452345f7c335d975b7df71a9506277a80b152a3370831bceebc10a94b2"}
Oct 30 01:37:56.198056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:56.197631    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"
Oct 30 01:37:56.199128 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:56.199099    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq" event={"ID":"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3","Type":"ContainerStarted","Data":"3c9aa7e858276e7717cf610b7d3e225bd60afd08210c23e59cca1c5798920370"}
Oct 30 01:37:56.199258 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:56.199224    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"
Oct 30 01:37:56.252673 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:56.252624    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c" podStartSLOduration=1.855326748 podStartE2EDuration="2.252612195s" totalImagesPullingTime="397.285447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:56.25178669 +0000 UTC m=+3770.108694445" watchObservedRunningTime="2026-10-30 01:37:56.252612195 +0000 UTC m=+3770.109519949"
Oct 30 01:37:56.305980 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:56.305932    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq" podStartSLOduration=1.923557564 podStartE2EDuration="2.305916321s" totalImagesPullingTime="382.358757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:56.292918884 +0000 UTC m=+3770.149826639" watchObservedRunningTime="2026-10-30 01:37:56.305916321 +0000 UTC m=+3770.162824072"
Oct 30 01:37:59.028139 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:59.028103    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx"]
Oct 30 01:37:59.039834 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:37:59.039808    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnddx"]
Oct 30 01:38:00.259904 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:00.259868    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3c063d2-e917-447f-94c7-1ff6cd9f1e7e" path="/var/lib/kubelet/pods/f3c063d2-e917-447f-94c7-1ff6cd9f1e7e/volumes"
Oct 30 01:38:02.208897 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:02.208867    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-h6g9c"
Oct 30 01:38:02.209385 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:02.209364    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"
Oct 30 01:38:02.283853 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:02.283817    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"]
Oct 30 01:38:02.284160 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:02.284134    6099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq" podUID="5d9b5b01-f0d1-428c-b005-1fc0aabda0d3" containerName="maas-api" containerID="cri-o://3c9aa7e858276e7717cf610b7d3e225bd60afd08210c23e59cca1c5798920370" gracePeriod=30
Oct 30 01:38:02.532364 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:02.532337    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"
Oct 30 01:38:02.634613 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:02.634578    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-maas-api-tls\") pod \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\" (UID: \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\") "
Oct 30 01:38:02.634863 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:02.634723    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-kube-api-access-mp9hl\" (UniqueName: \"kubernetes.io/projected/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-kube-api-access-mp9hl\") pod \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\" (UID: \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\") "
Oct 30 01:38:02.634863 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:02.634764    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-metrics-certs\") pod \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\" (UID: \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\") "
Oct 30 01:38:02.634863 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:02.634809    6099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-openshift-service-ca\") pod \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\" (UID: \"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3\") "
Oct 30 01:38:02.635255 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:02.635233    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-openshift-service-ca" pod "5d9b5b01-f0d1-428c-b005-1fc0aabda0d3" (UID: "5d9b5b01-f0d1-428c-b005-1fc0aabda0d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:38:02.636982 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:02.636960    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-metrics-certs" pod "5d9b5b01-f0d1-428c-b005-1fc0aabda0d3" (UID: "5d9b5b01-f0d1-428c-b005-1fc0aabda0d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:38:02.637329 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:02.637304    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-kube-api-access-mp9hl" pod "5d9b5b01-f0d1-428c-b005-1fc0aabda0d3" (UID: "5d9b5b01-f0d1-428c-b005-1fc0aabda0d3"). InnerVolumeSpecName "kube-api-access-mp9hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:38:02.637431 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:02.637304    6099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-maas-api-tls" pod "5d9b5b01-f0d1-428c-b005-1fc0aabda0d3" (UID: "5d9b5b01-f0d1-428c-b005-1fc0aabda0d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:38:02.736043 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:02.736011    6099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-maas-api-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:38:02.736043 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:02.736038    6099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp9hl\" (UniqueName: \"kubernetes.io/projected/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-kube-api-access-mp9hl\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:38:02.736043 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:02.736049    6099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-metrics-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:38:02.736271 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:02.736058    6099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3-openshift-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\""
Oct 30 01:38:02.767624 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:02.767556    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:38:03.236576 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:03.236517    6099 generic.go:356] "Generic (PLEG): container finished" podID="5d9b5b01-f0d1-428c-b005-1fc0aabda0d3" containerID="3c9aa7e858276e7717cf610b7d3e225bd60afd08210c23e59cca1c5798920370" exitCode=0
Oct 30 01:38:03.236576 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:03.236578    6099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"
Oct 30 01:38:03.237206 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:03.236586    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq" event={"ID":"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3","Type":"ContainerDied","Data":"3c9aa7e858276e7717cf610b7d3e225bd60afd08210c23e59cca1c5798920370"}
Oct 30 01:38:03.237206 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:03.236674    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-nkchq" event={"ID":"5d9b5b01-f0d1-428c-b005-1fc0aabda0d3","Type":"ContainerDied","Data":"a16f1eecf904104f40fc0448c8634700dde025810266cd3dc40122d6603ebb93"}
Oct 30 01:38:03.237206 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:03.236764    6099 scope.go:122] "RemoveContainer" containerID="3c9aa7e858276e7717cf610b7d3e225bd60afd08210c23e59cca1c5798920370"
Oct 30 01:38:03.248989 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:03.248968    6099 scope.go:122] "RemoveContainer" containerID="3c9aa7e858276e7717cf610b7d3e225bd60afd08210c23e59cca1c5798920370"
Oct 30 01:38:03.249275 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:38:03.249249    6099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c9aa7e858276e7717cf610b7d3e225bd60afd08210c23e59cca1c5798920370\": container with ID starting with 3c9aa7e858276e7717cf610b7d3e225bd60afd08210c23e59cca1c5798920370 not found: ID does not exist" containerID="3c9aa7e858276e7717cf610b7d3e225bd60afd08210c23e59cca1c5798920370"
Oct 30 01:38:03.249343 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:03.249287    6099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c9aa7e858276e7717cf610b7d3e225bd60afd08210c23e59cca1c5798920370"} err="failed to get container status \"3c9aa7e858276e7717cf610b7d3e225bd60afd08210c23e59cca1c5798920370\": rpc error: code = NotFound desc = could not find container \"3c9aa7e858276e7717cf610b7d3e225bd60afd08210c23e59cca1c5798920370\": container with ID starting with 3c9aa7e858276e7717cf610b7d3e225bd60afd08210c23e59cca1c5798920370 not found: ID does not exist"
Oct 30 01:38:03.290017 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:03.289980    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"]
Oct 30 01:38:03.300542 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:03.300511    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-nkchq"]
Oct 30 01:38:04.259400 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:04.259359    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d9b5b01-f0d1-428c-b005-1fc0aabda0d3" path="/var/lib/kubelet/pods/5d9b5b01-f0d1-428c-b005-1fc0aabda0d3/volumes"
Oct 30 01:38:08.639388 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:08.639353    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:38:16.163663 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:16.163631    6099 scope.go:122] "RemoveContainer" containerID="091ca9988b3888704afadf32f8d60b992b795ce99cb7ea21a534744acc64541c"
Oct 30 01:38:16.630409 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:38:16.630360    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:38:24.449236 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:24.449155    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:38:46.629262 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:38:46.629222    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:38:52.456401 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:52.456365    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2mqnz"]
Oct 30 01:38:52.654397 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:38:52.654361    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68f8ff9ccb-jznqk_78c5e37d-698d-492a-9870-f219f4d5beca/authorino/0.log"
Oct 30 01:39:02.923752 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:02.923716    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-84479857d9-2xt9z_5c6a2a03-fda8-46b2-9246-4d77746c9421/manager/0.log"
Oct 30 01:39:04.345004 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:04.344973    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-lbcfw_938fce0b-97c1-4d08-9565-f92cd7e064c7/manager/0.log"
Oct 30 01:39:05.060512 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:05.060485    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85c8f7c-k29rq_0fade35d-c453-4e88-a630-605cf4a7a364/copy-manifests/0.log"
Oct 30 01:39:05.090296 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:05.090271    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85c8f7c-k29rq_0fade35d-c453-4e88-a630-605cf4a7a364/manager/0.log"
Oct 30 01:39:06.471259 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:06.471231    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-h6g9c_9aa787cc-9150-483f-b4bf-968feda47917/maas-api/0.log"
Oct 30 01:39:06.845631 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:06.845598    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-35ff1eed-swmnh_d1362a15-d730-483f-bd3d-b83738bdf704/revoke-keys/0.log"
Oct 30 01:39:07.021760 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:07.021732    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-plsxg_c377dff7-9004-41c2-b1bf-1107b2ab617c/revoke-keys/0.log"
Oct 30 01:39:07.222961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:07.222889    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-pppmb_5b7f095c-42e2-41a2-8789-a8838ecffe10/revoke-keys/0.log"
Oct 30 01:39:07.757126 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:07.757102    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-582nz_de9406fe-652e-4378-99a9-310525a6ad78/revoke-keys/0.log"
Oct 30 01:39:07.941463 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:07.941431    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-fspp5_eb57fdac-79f5-42e4-9722-6840fed8ccdc/revoke-keys/0.log"
Oct 30 01:39:09.780151 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:09.780119    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68f8ff9ccb-jznqk_78c5e37d-698d-492a-9870-f219f4d5beca/authorino/0.log"
Oct 30 01:39:10.404029 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:10.403997    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8qnjq_85eff882-1ab7-4def-92d7-47c8135c44c9/kuadrant-console-plugin/0.log"
Oct 30 01:39:10.820576 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:10.820549    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2mqnz_d7baecc5-e3d8-46e8-9ec1-883d47e82c85/limitador/0.log"
Oct 30 01:39:12.120844 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:12.120816    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64d5b5b7d7-nttd5_3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270/kube-auth-proxy/0.log"
Oct 30 01:39:12.856637 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:12.856613    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-7nffr_1f88f534-38ba-41ff-b2e5-b7335f2f1fb2/istio-proxy/0.log"
Oct 30 01:39:13.821783 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:13.821751    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-pc67m_d77598a0-d6b3-421a-bd3d-5b13591ad686/istio-proxy/0.log"
Oct 30 01:39:14.210984 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:14.210902    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2_53e1681f-ad08-4174-86a2-261d28114fb8/istio-proxy/0.log"
Oct 30 01:39:14.566608 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:14.566579    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rcxc5_1569a420-ef8b-4ca9-86c1-cde78845b4b6/payload-processing/0.log"
Oct 30 01:39:16.066622 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:16.066589    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-hknzb_4f0e2175-e7cb-48f7-b2ac-1c18e1890cde/main/0.log"
Oct 30 01:39:16.191961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:16.191933    6099 scope.go:122] "RemoveContainer" containerID="bab2e52fc5f2c01a889ef7ef0065df7e36e92c54b5c7ab80ca09ab65bb06d80e"
Oct 30 01:39:16.443568 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:16.443488    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm_d94d18a3-84f0-4225-bf91-19ce9a058574/main/0.log"
Oct 30 01:39:16.628568 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:39:16.628532    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:39:16.636626 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:16.636606    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv_5c8179b0-96f4-4647-b926-5969b4afecd4/main/0.log"
Oct 30 01:39:17.689167 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:17.689133    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68f8ff9ccb-jznqk_78c5e37d-698d-492a-9870-f219f4d5beca/authorino/0.log"
Oct 30 01:39:26.511173 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:26.511140    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-84479857d9-2xt9z_5c6a2a03-fda8-46b2-9246-4d77746c9421/manager/0.log"
Oct 30 01:39:27.781533 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:27.781504    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-lbcfw_938fce0b-97c1-4d08-9565-f92cd7e064c7/manager/0.log"
Oct 30 01:39:28.458160 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:28.458130    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85c8f7c-k29rq_0fade35d-c453-4e88-a630-605cf4a7a364/copy-manifests/0.log"
Oct 30 01:39:28.486153 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:28.486127    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85c8f7c-k29rq_0fade35d-c453-4e88-a630-605cf4a7a364/manager/0.log"
Oct 30 01:39:29.613272 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:29.613243    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-h6g9c_9aa787cc-9150-483f-b4bf-968feda47917/maas-api/0.log"
Oct 30 01:39:29.991197 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:29.991129    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-35ff1eed-swmnh_d1362a15-d730-483f-bd3d-b83738bdf704/revoke-keys/0.log"
Oct 30 01:39:30.171489 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:30.171460    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-plsxg_c377dff7-9004-41c2-b1bf-1107b2ab617c/revoke-keys/0.log"
Oct 30 01:39:30.346120 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:30.346084    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-pppmb_5b7f095c-42e2-41a2-8789-a8838ecffe10/revoke-keys/0.log"
Oct 30 01:39:30.888123 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:30.888091    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-582nz_de9406fe-652e-4378-99a9-310525a6ad78/revoke-keys/0.log"
Oct 30 01:39:31.054175 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:31.054144    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-fspp5_eb57fdac-79f5-42e4-9722-6840fed8ccdc/revoke-keys/0.log"
Oct 30 01:39:32.535268 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:32.535237    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68f8ff9ccb-jznqk_78c5e37d-698d-492a-9870-f219f4d5beca/authorino/0.log"
Oct 30 01:39:33.173243 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:33.173212    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8qnjq_85eff882-1ab7-4def-92d7-47c8135c44c9/kuadrant-console-plugin/0.log"
Oct 30 01:39:33.635566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:33.635539    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2mqnz_d7baecc5-e3d8-46e8-9ec1-883d47e82c85/limitador/0.log"
Oct 30 01:39:34.789964 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:34.789939    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64d5b5b7d7-nttd5_3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270/kube-auth-proxy/0.log"
Oct 30 01:39:35.558185 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:35.558156    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-7nffr_1f88f534-38ba-41ff-b2e5-b7335f2f1fb2/istio-proxy/0.log"
Oct 30 01:39:36.517082 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:36.517049    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-pc67m_d77598a0-d6b3-421a-bd3d-5b13591ad686/istio-proxy/0.log"
Oct 30 01:39:36.906604 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:36.906574    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2_53e1681f-ad08-4174-86a2-261d28114fb8/istio-proxy/0.log"
Oct 30 01:39:37.281036 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:37.280943    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rcxc5_1569a420-ef8b-4ca9-86c1-cde78845b4b6/payload-processing/0.log"
Oct 30 01:39:38.425251 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:38.425221    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-hknzb_4f0e2175-e7cb-48f7-b2ac-1c18e1890cde/main/0.log"
Oct 30 01:39:38.816049 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:38.816011    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kqzwm_d94d18a3-84f0-4225-bf91-19ce9a058574/main/0.log"
Oct 30 01:39:39.004428 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:39.004358    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-d9dfv_5c8179b0-96f4-4647-b926-5969b4afecd4/main/0.log"
Oct 30 01:39:46.629451 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:39:46.629415    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:39:48.028741 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:48.028647    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5"]
Oct 30 01:39:48.040648 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:48.040623    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fspp5"]
Oct 30 01:39:48.267127 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:39:48.267092    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb57fdac-79f5-42e4-9722-6840fed8ccdc" path="/var/lib/kubelet/pods/eb57fdac-79f5-42e4-9722-6840fed8ccdc/volumes"
Oct 30 01:40:04.028541 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:40:04.028505    6099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz"]
Oct 30 01:40:04.040189 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:40:04.040163    6099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-582nz"]
Oct 30 01:40:04.260415 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:40:04.260380    6099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de9406fe-652e-4378-99a9-310525a6ad78" path="/var/lib/kubelet/pods/de9406fe-652e-4378-99a9-310525a6ad78/volumes"
Oct 30 01:40:07.044918 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:40:07.044887    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 01:40:07.059632 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:40:07.059606    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcqkc_070879ac-09e8-4fdb-b55f-e52a9998552f/ovn-acl-logging/0.log"
Oct 30 01:40:10.858180 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:40:10.858147    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rcxc5_1569a420-ef8b-4ca9-86c1-cde78845b4b6/payload-processing/0.log"
Oct 30 01:40:11.048826 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:40:11.048799    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rcxc5_1569a420-ef8b-4ca9-86c1-cde78845b4b6/payload-processing/0.log"
Oct 30 01:40:16.629700 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:40:16.629659    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:40:46.629011 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:40:46.628973    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:40:51.308227 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:40:51.308199    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rcxc5_1569a420-ef8b-4ca9-86c1-cde78845b4b6/payload-processing/0.log"
Oct 30 01:40:51.473601 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:40:51.473574    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rcxc5_1569a420-ef8b-4ca9-86c1-cde78845b4b6/payload-processing/0.log"
Oct 30 01:40:56.399980 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:40:56.399949    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7vsmg_b7f00b4a-044e-49ab-8f43-4351590692e8/global-pull-secret-syncer/0.log"
Oct 30 01:40:57.365887 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:40:57.365859    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v6rdh_47d34735-e4b1-4073-b6de-98a9a720432b/konnectivity-agent/0.log"
Oct 30 01:40:57.692130 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:40:57.692040    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-245.ec2.internal_f8c3a9edbd8cbf03175beab8c4db7242/haproxy/0.log"
Oct 30 01:41:03.831011 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:03.830978    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68f8ff9ccb-jznqk_78c5e37d-698d-492a-9870-f219f4d5beca/authorino/0.log"
Oct 30 01:41:04.183905 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:04.183838    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8qnjq_85eff882-1ab7-4def-92d7-47c8135c44c9/kuadrant-console-plugin/0.log"
Oct 30 01:41:04.461060 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:04.460984    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2mqnz_d7baecc5-e3d8-46e8-9ec1-883d47e82c85/limitador/0.log"
Oct 30 01:41:06.068106 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.068076    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8ff948dc-8d7a-435f-83bd-7a6a461c832a/alertmanager/0.log"
Oct 30 01:41:06.193357 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.193329    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8ff948dc-8d7a-435f-83bd-7a6a461c832a/config-reloader/0.log"
Oct 30 01:41:06.307331 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.307302    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8ff948dc-8d7a-435f-83bd-7a6a461c832a/kube-rbac-proxy-web/0.log"
Oct 30 01:41:06.417561 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.417494    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8ff948dc-8d7a-435f-83bd-7a6a461c832a/kube-rbac-proxy/0.log"
Oct 30 01:41:06.526586 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.526560    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8ff948dc-8d7a-435f-83bd-7a6a461c832a/kube-rbac-proxy-metric/0.log"
Oct 30 01:41:06.630653 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.630620    6099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"]
Oct 30 01:41:06.631129 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.631113    6099 cpu_manager.go:414] "containerMap: removing container" podUID="5d9b5b01-f0d1-428c-b005-1fc0aabda0d3" containerName="maas-api"
Oct 30 01:41:06.631191 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.631131    6099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9b5b01-f0d1-428c-b005-1fc0aabda0d3" containerName="maas-api"
Oct 30 01:41:06.631233 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.631192    6099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9b5b01-f0d1-428c-b005-1fc0aabda0d3" containerName="maas-api"
Oct 30 01:41:06.634952 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.634935    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:06.642566 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.642545    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8ff948dc-8d7a-435f-83bd-7a6a461c832a/prom-label-proxy/0.log"
Oct 30 01:41:06.653985 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.653956    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"]
Oct 30 01:41:06.663961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.663940    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8hpkt\"/\"openshift-service-ca.crt\""
Oct 30 01:41:06.663961 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.663945    6099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8hpkt\"/\"kube-root-ca.crt\""
Oct 30 01:41:06.664158 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.664031    6099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8hpkt\"/\"default-dockercfg-mlw7p\""
Oct 30 01:41:06.712274 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.712209    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/870fe220-9806-4131-868e-c42d24e5443a-proc\") pod \"perf-node-gather-daemonset-5j2hq\" (UID: \"870fe220-9806-4131-868e-c42d24e5443a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:06.712274 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.712249    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr7kz\" (UniqueName: \"kubernetes.io/projected/870fe220-9806-4131-868e-c42d24e5443a-kube-api-access-rr7kz\") pod \"perf-node-gather-daemonset-5j2hq\" (UID: \"870fe220-9806-4131-868e-c42d24e5443a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:06.712440 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.712319    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/870fe220-9806-4131-868e-c42d24e5443a-podres\") pod \"perf-node-gather-daemonset-5j2hq\" (UID: \"870fe220-9806-4131-868e-c42d24e5443a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:06.712483 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.712469    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/870fe220-9806-4131-868e-c42d24e5443a-sys\") pod \"perf-node-gather-daemonset-5j2hq\" (UID: \"870fe220-9806-4131-868e-c42d24e5443a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:06.712523 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.712514    6099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/870fe220-9806-4131-868e-c42d24e5443a-lib-modules\") pod \"perf-node-gather-daemonset-5j2hq\" (UID: \"870fe220-9806-4131-868e-c42d24e5443a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:06.763317 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.763292    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8ff948dc-8d7a-435f-83bd-7a6a461c832a/init-config-reloader/0.log"
Oct 30 01:41:06.813808 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.813780    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/870fe220-9806-4131-868e-c42d24e5443a-proc\") pod \"perf-node-gather-daemonset-5j2hq\" (UID: \"870fe220-9806-4131-868e-c42d24e5443a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:06.813976 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.813822    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr7kz\" (UniqueName: \"kubernetes.io/projected/870fe220-9806-4131-868e-c42d24e5443a-kube-api-access-rr7kz\") pod \"perf-node-gather-daemonset-5j2hq\" (UID: \"870fe220-9806-4131-868e-c42d24e5443a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:06.813976 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.813906    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/870fe220-9806-4131-868e-c42d24e5443a-proc\") pod \"perf-node-gather-daemonset-5j2hq\" (UID: \"870fe220-9806-4131-868e-c42d24e5443a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:06.813976 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.813945    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/870fe220-9806-4131-868e-c42d24e5443a-podres\") pod \"perf-node-gather-daemonset-5j2hq\" (UID: \"870fe220-9806-4131-868e-c42d24e5443a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:06.814126 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.814032    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/870fe220-9806-4131-868e-c42d24e5443a-podres\") pod \"perf-node-gather-daemonset-5j2hq\" (UID: \"870fe220-9806-4131-868e-c42d24e5443a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:06.814126 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.814079    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/870fe220-9806-4131-868e-c42d24e5443a-sys\") pod \"perf-node-gather-daemonset-5j2hq\" (UID: \"870fe220-9806-4131-868e-c42d24e5443a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:06.814126 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.814120    6099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/870fe220-9806-4131-868e-c42d24e5443a-lib-modules\") pod \"perf-node-gather-daemonset-5j2hq\" (UID: \"870fe220-9806-4131-868e-c42d24e5443a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:06.814279 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.814206    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/870fe220-9806-4131-868e-c42d24e5443a-lib-modules\") pod \"perf-node-gather-daemonset-5j2hq\" (UID: \"870fe220-9806-4131-868e-c42d24e5443a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:06.814279 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.814232    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/870fe220-9806-4131-868e-c42d24e5443a-sys\") pod \"perf-node-gather-daemonset-5j2hq\" (UID: \"870fe220-9806-4131-868e-c42d24e5443a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:06.838151 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.838128    6099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr7kz\" (UniqueName: \"kubernetes.io/projected/870fe220-9806-4131-868e-c42d24e5443a-kube-api-access-rr7kz\") pod \"perf-node-gather-daemonset-5j2hq\" (UID: \"870fe220-9806-4131-868e-c42d24e5443a\") " pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:06.947769 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:06.947747    6099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:07.019842 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:07.019818    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-pcmnz_bc8e4d97-4c7f-450d-961c-528c992a29b2/kube-state-metrics/0.log"
Oct 30 01:41:07.124067 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:07.124031    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-pcmnz_bc8e4d97-4c7f-450d-961c-528c992a29b2/kube-rbac-proxy-main/0.log"
Oct 30 01:41:07.241512 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:07.241484    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-pcmnz_bc8e4d97-4c7f-450d-961c-528c992a29b2/kube-rbac-proxy-self/0.log"
Oct 30 01:41:07.253801 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:07.253780    6099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"]
Oct 30 01:41:07.255900 ip-10-0-130-245 kubenswrapper[6099]: W1030 01:41:07.255871    6099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod870fe220_9806_4131_868e_c42d24e5443a.slice/crio-b13507daa9d148a4bf3bb5a7c1a79df2ef0c4990d77746b58ae06f84db1e4907 WatchSource:0}: Error finding container b13507daa9d148a4bf3bb5a7c1a79df2ef0c4990d77746b58ae06f84db1e4907: Status 404 returned error can't find the container with id b13507daa9d148a4bf3bb5a7c1a79df2ef0c4990d77746b58ae06f84db1e4907
Oct 30 01:41:07.363949 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:07.363927    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7c55cf7f78-h65h8_b8c48d07-5599-4a74-94d4-d6488bde2827/metrics-server/0.log"
Oct 30 01:41:07.479531 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:07.479505    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-tljgj_12bf7b6a-fd62-4921-910d-f9fff2b347d1/monitoring-plugin/0.log"
Oct 30 01:41:08.247103 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:08.247068    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq" event={"ID":"870fe220-9806-4131-868e-c42d24e5443a","Type":"ContainerStarted","Data":"f3237b32d7af628dd213218ad7848790ae92a472220b98584fe96b31d6aab17a"}
Oct 30 01:41:08.247504 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:08.247146    6099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq" event={"ID":"870fe220-9806-4131-868e-c42d24e5443a","Type":"ContainerStarted","Data":"b13507daa9d148a4bf3bb5a7c1a79df2ef0c4990d77746b58ae06f84db1e4907"}
Oct 30 01:41:08.247504 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:08.247216    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:08.287201 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:08.287152    6099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq" podStartSLOduration=2.287139588 podStartE2EDuration="2.287139588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:08.286046001 +0000 UTC m=+3962.142953758" watchObservedRunningTime="2026-10-30 01:41:08.287139588 +0000 UTC m=+3962.144047343"
Oct 30 01:41:08.332556 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:08.332532    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wb627_f53ec687-40dd-4358-bc05-83d064d9d082/node-exporter/0.log"
Oct 30 01:41:08.452407 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:08.452385    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wb627_f53ec687-40dd-4358-bc05-83d064d9d082/kube-rbac-proxy/0.log"
Oct 30 01:41:08.567547 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:08.567521    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wb627_f53ec687-40dd-4358-bc05-83d064d9d082/init-textfile/0.log"
Oct 30 01:41:08.684338 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:08.684310    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-4qfxh_962a4e53-88b6-4866-abc5-108b6d5ce784/kube-rbac-proxy-main/0.log"
Oct 30 01:41:08.798211 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:08.798185    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-4qfxh_962a4e53-88b6-4866-abc5-108b6d5ce784/kube-rbac-proxy-self/0.log"
Oct 30 01:41:08.914604 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:08.914540    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-4qfxh_962a4e53-88b6-4866-abc5-108b6d5ce784/openshift-state-metrics/0.log"
Oct 30 01:41:09.055179 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:09.055152    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7ab060b7-ab03-4bd9-b917-9a4945a0b614/prometheus/0.log"
Oct 30 01:41:09.178261 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:09.178195    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7ab060b7-ab03-4bd9-b917-9a4945a0b614/config-reloader/0.log"
Oct 30 01:41:09.294923 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:09.294899    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7ab060b7-ab03-4bd9-b917-9a4945a0b614/thanos-sidecar/0.log"
Oct 30 01:41:09.419346 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:09.419321    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7ab060b7-ab03-4bd9-b917-9a4945a0b614/kube-rbac-proxy-web/0.log"
Oct 30 01:41:09.534241 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:09.534152    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7ab060b7-ab03-4bd9-b917-9a4945a0b614/kube-rbac-proxy/0.log"
Oct 30 01:41:09.655318 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:09.655292    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7ab060b7-ab03-4bd9-b917-9a4945a0b614/kube-rbac-proxy-thanos/0.log"
Oct 30 01:41:09.797348 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:09.797322    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7ab060b7-ab03-4bd9-b917-9a4945a0b614/init-config-reloader/0.log"
Oct 30 01:41:09.933619 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:09.933593    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-ls4dj_21be6874-6138-4a00-9c6e-aa1841eb3230/prometheus-operator/0.log"
Oct 30 01:41:10.045844 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:10.045815    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-ls4dj_21be6874-6138-4a00-9c6e-aa1841eb3230/kube-rbac-proxy/0.log"
Oct 30 01:41:10.289945 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:10.289922    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6ddf586968-ml4vv_ddb2dce5-50be-42dc-88f4-ecb4d6a29d28/telemeter-client/0.log"
Oct 30 01:41:10.397273 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:10.397246    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6ddf586968-ml4vv_ddb2dce5-50be-42dc-88f4-ecb4d6a29d28/reload/0.log"
Oct 30 01:41:10.510013 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:10.509980    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6ddf586968-ml4vv_ddb2dce5-50be-42dc-88f4-ecb4d6a29d28/kube-rbac-proxy/0.log"
Oct 30 01:41:10.631402 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:10.631336    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-676949fd44-65zsj_c5282344-9beb-485c-9525-6b09bdda0804/thanos-query/0.log"
Oct 30 01:41:10.744862 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:10.744817    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-676949fd44-65zsj_c5282344-9beb-485c-9525-6b09bdda0804/kube-rbac-proxy-web/0.log"
Oct 30 01:41:10.854174 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:10.854150    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-676949fd44-65zsj_c5282344-9beb-485c-9525-6b09bdda0804/kube-rbac-proxy/0.log"
Oct 30 01:41:10.963296 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:10.963229    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-676949fd44-65zsj_c5282344-9beb-485c-9525-6b09bdda0804/prom-label-proxy/0.log"
Oct 30 01:41:11.079141 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:11.079110    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-676949fd44-65zsj_c5282344-9beb-485c-9525-6b09bdda0804/kube-rbac-proxy-rules/0.log"
Oct 30 01:41:11.187345 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:11.187319    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-676949fd44-65zsj_c5282344-9beb-485c-9525-6b09bdda0804/kube-rbac-proxy-metrics/0.log"
Oct 30 01:41:14.263475 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:14.263448    6099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8hpkt/perf-node-gather-daemonset-5j2hq"
Oct 30 01:41:16.224151 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:16.224122    6099 scope.go:122] "RemoveContainer" containerID="2c9c9fc4f8b27890b331b115a71156f4b85be43cb00ef96e8e000f5b80b93a61"
Oct 30 01:41:16.233323 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:16.233306    6099 scope.go:122] "RemoveContainer" containerID="dfa8e26bf7237c3c98dfb59baf2d165640e7f66ba54eda363f2b6dd85b5655c8"
Oct 30 01:41:16.628935 ip-10-0-130-245 kubenswrapper[6099]: E1030 01:41:16.628900    6099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wvmfk" containerName="extractor"
Oct 30 01:41:18.638966 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:18.638897    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rwwwf_70c3a600-551a-4ff4-8fbe-e9a2aa1172e3/dns/0.log"
Oct 30 01:41:18.755293 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:18.755266    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rwwwf_70c3a600-551a-4ff4-8fbe-e9a2aa1172e3/kube-rbac-proxy/0.log"
Oct 30 01:41:19.293302 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:19.293279    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b2bnk_b45c13d7-29c0-4054-9b7d-fd33ecec9cf7/dns-node-resolver/0.log"
Oct 30 01:41:20.410137 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:20.410102    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64d5b5b7d7-nttd5_3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270/kube-auth-proxy/0.log"
Oct 30 01:41:20.676687 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:20.676622    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-7nffr_1f88f534-38ba-41ff-b2e5-b7335f2f1fb2/istio-proxy/0.log"
Oct 30 01:41:20.876556 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:20.876526    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d9lhd_5975a832-beb3-4e76-98ad-aef36a2ccce2/node-ca/0.log"
Oct 30 01:41:21.070639 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:21.070611    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-pc67m_d77598a0-d6b3-421a-bd3d-5b13591ad686/istio-proxy/0.log"
Oct 30 01:41:21.203876 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:21.203849    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-zpcs2_53e1681f-ad08-4174-86a2-261d28114fb8/istio-proxy/0.log"
Oct 30 01:41:21.336056 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:21.335984    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rcxc5_1569a420-ef8b-4ca9-86c1-cde78845b4b6/payload-processing/0.log"
Oct 30 01:41:24.182788 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:24.182760    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64d5b5b7d7-nttd5_3a5daaf9-d4a9-4d5c-bb8f-b3ea12c55270/kube-auth-proxy/0.log"
Oct 30 01:41:24.651183 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:24.651156    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-7nffr_1f88f534-38ba-41ff-b2e5-b7335f2f1fb2/istio-proxy/0.log"
Oct 30 01:41:25.310443 ip-10-0-130-245 kubenswrapper[6099]: I1030 01:41:25.310411    6099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-pc67m_d77598a0-d6b3-421a-bd3d-5b13591ad686/istio-proxy/0.log"